2023-03-30 03:56:03,789 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e 2023-03-30 03:56:03,801 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-30 03:56:03,815 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-30 03:56:03,816 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/cluster_93f5cab8-c930-d024-2758-c8a21862fe6f, deleteOnExit=true 2023-03-30 03:56:03,816 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-30 03:56:03,817 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/test.cache.data in system properties and HBase conf 2023-03-30 03:56:03,817 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/hadoop.tmp.dir in system properties and HBase conf 2023-03-30 03:56:03,818 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/hadoop.log.dir in system properties and HBase conf 2023-03-30 03:56:03,818 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-30 03:56:03,819 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-30 03:56:03,819 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-30 03:56:03,920 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-30 03:56:04,250 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-30 03:56:04,254 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-30 03:56:04,254 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-30 03:56:04,255 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-30 03:56:04,255 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-30 03:56:04,255 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-30 03:56:04,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-30 03:56:04,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-30 03:56:04,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-30 03:56:04,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-30 03:56:04,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/nfs.dump.dir in system properties and HBase conf 2023-03-30 03:56:04,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/java.io.tmpdir in system properties and HBase conf 2023-03-30 03:56:04,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-30 03:56:04,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-30 03:56:04,258 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-30 03:56:04,702 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-30 03:56:04,706 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-30 03:56:04,962 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-30 03:56:05,136 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-30 03:56:05,156 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-30 03:56:05,193 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-30 03:56:05,233 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/java.io.tmpdir/Jetty_localhost_localdomain_35789_hdfs____.f6riog/webapp 2023-03-30 03:56:05,351 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:35789 2023-03-30 03:56:05,381 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-30 03:56:05,381 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-30 03:56:05,779 WARN [Listener at localhost.localdomain/36859] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-30 03:56:05,876 WARN [Listener at localhost.localdomain/36859] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-30 03:56:05,898 WARN [Listener at localhost.localdomain/36859] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-30 03:56:05,906 INFO [Listener at localhost.localdomain/36859] log.Slf4jLog(67): jetty-6.1.26 2023-03-30 03:56:05,912 INFO [Listener at localhost.localdomain/36859] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/java.io.tmpdir/Jetty_localhost_38221_datanode____mj6gyb/webapp 2023-03-30 03:56:06,018 INFO [Listener at localhost.localdomain/36859] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38221 2023-03-30 03:56:06,323 WARN [Listener at localhost.localdomain/37215] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-30 03:56:06,702 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1499d21f5c784684: Processing first storage report for DS-924760dc-bd35-460e-945e-8f79128675e3 from datanode 00825c34-6b27-4d8f-a0b1-9ef71ed73366 2023-03-30 03:56:06,704 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1499d21f5c784684: from storage DS-924760dc-bd35-460e-945e-8f79128675e3 node DatanodeRegistration(127.0.0.1:45009, datanodeUuid=00825c34-6b27-4d8f-a0b1-9ef71ed73366, infoPort=39127, infoSecurePort=0, ipcPort=37215, storageInfo=lv=-57;cid=testClusterID;nsid=1196327673;c=1680148564762), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-30 03:56:06,704 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1499d21f5c784684: Processing first storage report for DS-dea03d7e-533a-4c1f-a01d-58afcb883d45 from datanode 00825c34-6b27-4d8f-a0b1-9ef71ed73366 2023-03-30 03:56:06,704 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1499d21f5c784684: from storage DS-dea03d7e-533a-4c1f-a01d-58afcb883d45 node DatanodeRegistration(127.0.0.1:45009, datanodeUuid=00825c34-6b27-4d8f-a0b1-9ef71ed73366, infoPort=39127, infoSecurePort=0, ipcPort=37215, storageInfo=lv=-57;cid=testClusterID;nsid=1196327673;c=1680148564762), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-30 03:56:06,906 DEBUG [Listener at localhost.localdomain/37215] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e 2023-03-30 03:56:06,963 INFO [Listener at localhost.localdomain/37215] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/cluster_93f5cab8-c930-d024-2758-c8a21862fe6f/zookeeper_0, clientPort=58033, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/cluster_93f5cab8-c930-d024-2758-c8a21862fe6f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/cluster_93f5cab8-c930-d024-2758-c8a21862fe6f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-30 03:56:06,976 INFO [Listener at localhost.localdomain/37215] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58033 2023-03-30 03:56:06,984 INFO [Listener at localhost.localdomain/37215] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 03:56:06,986 INFO [Listener at localhost.localdomain/37215] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 03:56:07,668 INFO [Listener at localhost.localdomain/37215] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95 with version=8 2023-03-30 03:56:07,668 INFO [Listener at localhost.localdomain/37215] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/hbase-staging 2023-03-30 03:56:07,943 INFO [Listener at localhost.localdomain/37215] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-30 03:56:08,371 INFO [Listener at localhost.localdomain/37215] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-03-30 03:56:08,403 INFO [Listener at localhost.localdomain/37215] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-30 03:56:08,403 INFO [Listener at localhost.localdomain/37215] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-30 03:56:08,403 INFO [Listener at localhost.localdomain/37215] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-30 03:56:08,404 INFO [Listener at localhost.localdomain/37215] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-30 03:56:08,404 INFO [Listener at localhost.localdomain/37215] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-30 03:56:08,541 INFO [Listener at localhost.localdomain/37215] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-30 03:56:08,608 DEBUG [Listener at localhost.localdomain/37215] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-30 03:56:08,687 INFO [Listener at localhost.localdomain/37215] ipc.NettyRpcServer(120): Bind to /148.251.75.209:37051 2023-03-30 03:56:08,696 INFO [Listener at localhost.localdomain/37215] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 03:56:08,698 INFO [Listener at localhost.localdomain/37215] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 03:56:08,714 INFO [Listener at localhost.localdomain/37215] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37051 connecting to ZooKeeper ensemble=127.0.0.1:58033 2023-03-30 03:56:08,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:370510x0, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-30 03:56:08,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37051-0x100605eabcf0000 connected 2023-03-30 03:56:08,769 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-30 03:56:08,770 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-30 03:56:08,773 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-30 03:56:08,780 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37051 2023-03-30 03:56:08,780 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37051 2023-03-30 03:56:08,781 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37051 2023-03-30 03:56:08,781 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37051 2023-03-30 03:56:08,782 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37051 2023-03-30 03:56:08,790 INFO [Listener at localhost.localdomain/37215] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95, hbase.cluster.distributed=false 2023-03-30 03:56:08,852 INFO [Listener at localhost.localdomain/37215] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-03-30 03:56:08,852 INFO [Listener at localhost.localdomain/37215] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-30 03:56:08,852 INFO [Listener at localhost.localdomain/37215] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-30 03:56:08,852 INFO [Listener at localhost.localdomain/37215] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-30 03:56:08,852 INFO [Listener at localhost.localdomain/37215] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-30 03:56:08,852 INFO [Listener at localhost.localdomain/37215] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-30 03:56:08,856 INFO [Listener at localhost.localdomain/37215] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-30 03:56:08,859 INFO [Listener at localhost.localdomain/37215] ipc.NettyRpcServer(120): Bind to /148.251.75.209:41157 2023-03-30 03:56:08,861 INFO [Listener at localhost.localdomain/37215] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-30 03:56:08,866 DEBUG [Listener at localhost.localdomain/37215] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-30 03:56:08,867 INFO [Listener at localhost.localdomain/37215] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 03:56:08,869 INFO [Listener at localhost.localdomain/37215] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 03:56:08,871 INFO [Listener at localhost.localdomain/37215] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41157 connecting to ZooKeeper ensemble=127.0.0.1:58033 2023-03-30 03:56:08,874 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:411570x0, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-30 03:56:08,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41157-0x100605eabcf0001 connected 2023-03-30 03:56:08,875 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-30 03:56:08,876 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-30 03:56:08,877 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-30 03:56:08,877 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41157 2023-03-30 03:56:08,878 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41157 2023-03-30 03:56:08,878 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41157 2023-03-30 03:56:08,881 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41157 2023-03-30 03:56:08,881 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41157 2023-03-30 03:56:08,885 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:56:08,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-30 03:56:08,893 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:56:08,923 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-30 03:56:08,923 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-30 03:56:08,923 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:56:08,924 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-30 03:56:08,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-30 03:56:08,925 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,37051,1680148567795 from backup master directory 2023-03-30 03:56:08,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:56:08,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-30 03:56:08,929 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-30 03:56:08,929 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:56:08,931 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-30 03:56:08,932 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-30 03:56:09,405 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/hbase.id with ID: f887c1fe-e244-4c62-bed0-3afd914090e5 2023-03-30 03:56:09,450 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 03:56:09,467 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:56:09,503 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x5adf05a0 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:09,532 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26abf6cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:09,553 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:56:09,555 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-30 03:56:09,571 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-30 03:56:09,571 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-30 03:56:09,573 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:09,576 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:09,577 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-30 03:56:09,615 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store-tmp 2023-03-30 03:56:09,641 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:09,641 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-30 03:56:09,642 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 03:56:09,642 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 03:56:09,642 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-30 03:56:09,642 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 03:56:09,642 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 03:56:09,642 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-30 03:56:09,644 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/WALs/jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:56:09,668 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37051%2C1680148567795, suffix=, logDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/WALs/jenkins-hbase20.apache.org,37051,1680148567795, archiveDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/oldWALs, maxLogs=10 2023-03-30 03:56:09,727 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45009,DS-924760dc-bd35-460e-945e-8f79128675e3,DISK] 2023-03-30 03:56:09,736 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:09,795 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/WALs/jenkins-hbase20.apache.org,37051,1680148567795/jenkins-hbase20.apache.org%2C37051%2C1680148567795.1680148569680 2023-03-30 03:56:09,796 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45009,DS-924760dc-bd35-460e-945e-8f79128675e3,DISK]] 2023-03-30 03:56:09,796 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:56:09,796 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:09,799 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-30 03:56:09,800 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-30 03:56:09,853 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-30 03:56:09,860 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-30 03:56:09,881 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-30 03:56:09,894 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:09,900 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-30 03:56:09,902 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-30 03:56:09,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-30 03:56:09,924 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:56:09,925 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59108738, jitterRate=-0.11921116709709167}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-30 03:56:09,925 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-30 03:56:09,926 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-30 03:56:09,944 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-30 03:56:09,944 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-30 03:56:09,946 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-30 03:56:09,947 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-30 03:56:09,973 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-03-30 03:56:09,973 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-30 03:56:09,995 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-30 03:56:10,000 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-30 03:56:10,022 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-30 03:56:10,025 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-30 03:56:10,027 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-30 03:56:10,031 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-30 03:56:10,035 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-30 03:56:10,037 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:56:10,038 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-30 03:56:10,039 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-30 03:56:10,053 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-30 03:56:10,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-30 03:56:10,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-30 03:56:10,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:56:10,058 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,37051,1680148567795, sessionid=0x100605eabcf0000, setting cluster-up flag (Was=false) 2023-03-30 03:56:10,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:56:10,076 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-30 03:56:10,078 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:56:10,083 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:56:10,088 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-30 03:56:10,090 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:56:10,093 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.hbase-snapshot/.tmp 2023-03-30 03:56:10,194 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(952): ClusterId : f887c1fe-e244-4c62-bed0-3afd914090e5 2023-03-30 03:56:10,201 DEBUG [RS:0;jenkins-hbase20:41157] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-30 03:56:10,215 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-30 03:56:10,216 DEBUG [RS:0;jenkins-hbase20:41157] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-30 03:56:10,216 DEBUG [RS:0;jenkins-hbase20:41157] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-30 03:56:10,219 DEBUG [RS:0;jenkins-hbase20:41157] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-30 03:56:10,220 DEBUG [RS:0;jenkins-hbase20:41157] zookeeper.ReadOnlyZKClient(139): Connect 0x35a717e9 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:10,235 DEBUG [RS:0;jenkins-hbase20:41157] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b60ef2d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:10,236 DEBUG [RS:0;jenkins-hbase20:41157] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f5f7c99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-30 03:56:10,238 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-30 03:56:10,238 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-30 03:56:10,238 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-30 03:56:10,238 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-30 03:56:10,238 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-03-30 03:56:10,238 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,239 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-30 03:56:10,239 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,256 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680148600256 2023-03-30 03:56:10,259 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-30 03:56:10,269 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:41157 2023-03-30 03:56:10,275 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-30 03:56:10,275 INFO [RS:0;jenkins-hbase20:41157] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-30 03:56:10,275 INFO [RS:0;jenkins-hbase20:41157] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-30 03:56:10,275 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-30 03:56:10,275 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1023): About to register with Master. 2023-03-30 03:56:10,277 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-30 03:56:10,279 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,37051,1680148567795 with isa=jenkins-hbase20.apache.org/148.251.75.209:41157, startcode=1680148568851 2023-03-30 03:56:10,281 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-30 03:56:10,291 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-30 03:56:10,291 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-30 03:56:10,292 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-30 03:56:10,294 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-30 03:56:10,300 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,302 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-30 03:56:10,303 DEBUG [RS:0;jenkins-hbase20:41157] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-30 03:56:10,304 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-30 03:56:10,304 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-30 03:56:10,310 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-30 03:56:10,311 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-30 03:56:10,318 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680148570312,5,FailOnTimeoutGroup] 2023-03-30 03:56:10,322 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680148570318,5,FailOnTimeoutGroup] 2023-03-30 03:56:10,322 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,322 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-30 03:56:10,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,325 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,372 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36887, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-30 03:56:10,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:10,403 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95 2023-03-30 03:56:10,403 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:36859 2023-03-30 03:56:10,403 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-30 03:56:10,409 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-30 03:56:10,409 DEBUG [RS:0;jenkins-hbase20:41157] zookeeper.ZKUtil(162): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:10,409 WARN [RS:0;jenkins-hbase20:41157] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-30 03:56:10,410 INFO [RS:0;jenkins-hbase20:41157] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-30 03:56:10,411 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/WALs/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:10,413 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,41157,1680148568851] 2023-03-30 03:56:10,422 DEBUG [RS:0;jenkins-hbase20:41157] zookeeper.ZKUtil(162): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:10,434 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-30 03:56:10,443 INFO [RS:0;jenkins-hbase20:41157] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-30 03:56:10,466 INFO [RS:0;jenkins-hbase20:41157] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-30 03:56:10,469 INFO [RS:0;jenkins-hbase20:41157] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-30 03:56:10,470 INFO [RS:0;jenkins-hbase20:41157] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,471 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-30 03:56:10,478 INFO [RS:0;jenkins-hbase20:41157] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,478 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,479 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,479 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,479 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,479 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,479 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-30 03:56:10,480 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,480 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,480 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,480 DEBUG [RS:0;jenkins-hbase20:41157] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-30 03:56:10,486 INFO [RS:0;jenkins-hbase20:41157] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,487 INFO [RS:0;jenkins-hbase20:41157] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,487 INFO [RS:0;jenkins-hbase20:41157] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,506 INFO [RS:0;jenkins-hbase20:41157] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-30 03:56:10,509 INFO [RS:0;jenkins-hbase20:41157] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,41157,1680148568851-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:10,531 INFO [RS:0;jenkins-hbase20:41157] regionserver.Replication(203): jenkins-hbase20.apache.org,41157,1680148568851 started 2023-03-30 03:56:10,532 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,41157,1680148568851, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:41157, sessionid=0x100605eabcf0001 2023-03-30 03:56:10,532 DEBUG [RS:0;jenkins-hbase20:41157] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-30 03:56:10,532 DEBUG [RS:0;jenkins-hbase20:41157] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:10,532 DEBUG [RS:0;jenkins-hbase20:41157] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41157,1680148568851' 2023-03-30 03:56:10,532 DEBUG [RS:0;jenkins-hbase20:41157] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:10,533 DEBUG [RS:0;jenkins-hbase20:41157] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:10,534 DEBUG [RS:0;jenkins-hbase20:41157] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-30 03:56:10,534 DEBUG [RS:0;jenkins-hbase20:41157] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-30 03:56:10,534 DEBUG [RS:0;jenkins-hbase20:41157] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:10,534 DEBUG [RS:0;jenkins-hbase20:41157] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41157,1680148568851' 2023-03-30 03:56:10,534 DEBUG [RS:0;jenkins-hbase20:41157] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-30 03:56:10,535 DEBUG [RS:0;jenkins-hbase20:41157] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-30 03:56:10,535 DEBUG [RS:0;jenkins-hbase20:41157] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-30 03:56:10,535 INFO [RS:0;jenkins-hbase20:41157] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-30 03:56:10,536 INFO [RS:0;jenkins-hbase20:41157] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-30 03:56:10,650 INFO [RS:0;jenkins-hbase20:41157] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C41157%2C1680148568851, suffix=, logDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/WALs/jenkins-hbase20.apache.org,41157,1680148568851, archiveDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/oldWALs, maxLogs=32 2023-03-30 03:56:10,674 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45009,DS-924760dc-bd35-460e-945e-8f79128675e3,DISK] 2023-03-30 03:56:10,682 INFO [RS:0;jenkins-hbase20:41157] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/WALs/jenkins-hbase20.apache.org,41157,1680148568851/jenkins-hbase20.apache.org%2C41157%2C1680148568851.1680148570653 2023-03-30 03:56:10,682 DEBUG [RS:0;jenkins-hbase20:41157] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45009,DS-924760dc-bd35-460e-945e-8f79128675e3,DISK]] 2023-03-30 03:56:10,753 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-30 03:56:10,754 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-30 03:56:10,755 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95 2023-03-30 03:56:10,777 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:10,781 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-30 03:56:10,786 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/info 2023-03-30 03:56:10,787 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-30 03:56:10,789 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:10,789 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-30 03:56:10,793 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/rep_barrier 2023-03-30 03:56:10,794 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-30 03:56:10,795 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:10,796 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-30 03:56:10,799 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/table 2023-03-30 03:56:10,800 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-30 03:56:10,801 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:10,804 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740 2023-03-30 03:56:10,805 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740 2023-03-30 03:56:10,814 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:56:10,817 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-30 03:56:10,824 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:56:10,825 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67796912, jitterRate=0.010252714157104492}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:56:10,825 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-30 03:56:10,825 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-30 03:56:10,826 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-30 03:56:10,826 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-30 03:56:10,826 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-30 03:56:10,826 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-30 03:56:10,828 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-30 03:56:10,828 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-30 03:56:10,835 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-30 03:56:10,835 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-30 03:56:10,845 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-30 03:56:10,858 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-30 03:56:10,862 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-30 03:56:11,016 DEBUG [jenkins-hbase20:37051] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-30 03:56:11,020 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,41157,1680148568851, state=OPENING 2023-03-30 03:56:11,032 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-30 03:56:11,035 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:56:11,036 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-30 03:56:11,039 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:56:11,229 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:11,233 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-30 03:56:11,237 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37578, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-30 03:56:11,249 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-30 03:56:11,249 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-30 03:56:11,253 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C41157%2C1680148568851.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/WALs/jenkins-hbase20.apache.org,41157,1680148568851, archiveDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/oldWALs, maxLogs=32 2023-03-30 03:56:11,273 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45009,DS-924760dc-bd35-460e-945e-8f79128675e3,DISK] 2023-03-30 03:56:11,277 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/WALs/jenkins-hbase20.apache.org,41157,1680148568851/jenkins-hbase20.apache.org%2C41157%2C1680148568851.meta.1680148571255.meta 2023-03-30 03:56:11,278 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45009,DS-924760dc-bd35-460e-945e-8f79128675e3,DISK]] 2023-03-30 03:56:11,278 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:56:11,280 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-30 03:56:11,300 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-30 03:56:11,305 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-30 03:56:11,312 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-30 03:56:11,312 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:11,312 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-30 03:56:11,312 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-30 03:56:11,316 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-30 03:56:11,318 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/info 2023-03-30 03:56:11,318 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/info 2023-03-30 03:56:11,318 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-30 03:56:11,319 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:11,319 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-30 03:56:11,321 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/rep_barrier 2023-03-30 03:56:11,321 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/rep_barrier 2023-03-30 03:56:11,323 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-30 03:56:11,324 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:11,324 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-30 03:56:11,325 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/table 2023-03-30 03:56:11,326 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/table 2023-03-30 03:56:11,326 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-30 03:56:11,327 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:11,329 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740 2023-03-30 03:56:11,333 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740 2023-03-30 03:56:11,336 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:56:11,339 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-30 03:56:11,341 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70048037, jitterRate=0.0437970906496048}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:56:11,341 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-30 03:56:11,353 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680148571219 2023-03-30 03:56:11,375 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-30 03:56:11,376 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,41157,1680148568851, state=OPEN 2023-03-30 03:56:11,376 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-30 03:56:11,378 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-30 03:56:11,378 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-30 03:56:11,383 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-30 03:56:11,383 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,41157,1680148568851 in 339 msec 2023-03-30 03:56:11,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-30 03:56:11,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 539 msec 2023-03-30 03:56:11,397 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2530 sec 2023-03-30 03:56:11,397 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680148571397, completionTime=-1 2023-03-30 03:56:11,397 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-30 03:56:11,397 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-30 03:56:11,465 DEBUG [hconnection-0xce2ac86-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:11,475 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37586, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:11,497 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-30 03:56:11,497 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680148631497 2023-03-30 03:56:11,497 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680148691497 2023-03-30 03:56:11,497 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 99 msec 2023-03-30 03:56:11,521 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37051,1680148567795-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:11,521 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37051,1680148567795-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:11,522 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37051,1680148567795-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:11,523 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:37051, period=300000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:11,523 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-30 03:56:11,530 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-30 03:56:11,540 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-30 03:56:11,541 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-30 03:56:11,554 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-30 03:56:11,557 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 03:56:11,560 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 03:56:11,582 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:11,585 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0 empty. 2023-03-30 03:56:11,586 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:11,586 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-30 03:56:12,026 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-30 03:56:12,029 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4f7ceee8cd66bf2f0907e201f9289fa0, NAME => 'hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp 2023-03-30 03:56:12,441 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:12,442 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 4f7ceee8cd66bf2f0907e201f9289fa0, disabling compactions & flushes 2023-03-30 03:56:12,442 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:56:12,442 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:56:12,442 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. after waiting 0 ms 2023-03-30 03:56:12,442 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:56:12,442 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:56:12,442 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 4f7ceee8cd66bf2f0907e201f9289fa0: 2023-03-30 03:56:12,448 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 03:56:12,466 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680148572452"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148572452"}]},"ts":"1680148572452"} 2023-03-30 03:56:12,489 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 03:56:12,491 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 03:56:12,495 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148572491"}]},"ts":"1680148572491"} 2023-03-30 03:56:12,499 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-30 03:56:12,505 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4f7ceee8cd66bf2f0907e201f9289fa0, ASSIGN}] 2023-03-30 03:56:12,508 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4f7ceee8cd66bf2f0907e201f9289fa0, ASSIGN 2023-03-30 03:56:12,510 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=4f7ceee8cd66bf2f0907e201f9289fa0, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=false 2023-03-30 03:56:12,662 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4f7ceee8cd66bf2f0907e201f9289fa0, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:12,663 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680148572662"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148572662"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148572662"}]},"ts":"1680148572662"} 2023-03-30 03:56:12,673 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 4f7ceee8cd66bf2f0907e201f9289fa0, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:56:12,838 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:56:12,839 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4f7ceee8cd66bf2f0907e201f9289fa0, NAME => 'hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:56:12,840 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:12,841 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:12,841 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:12,841 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:12,843 INFO [StoreOpener-4f7ceee8cd66bf2f0907e201f9289fa0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:12,845 DEBUG [StoreOpener-4f7ceee8cd66bf2f0907e201f9289fa0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0/info 2023-03-30 03:56:12,845 DEBUG [StoreOpener-4f7ceee8cd66bf2f0907e201f9289fa0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0/info 2023-03-30 03:56:12,846 INFO [StoreOpener-4f7ceee8cd66bf2f0907e201f9289fa0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f7ceee8cd66bf2f0907e201f9289fa0 columnFamilyName info 2023-03-30 03:56:12,846 INFO [StoreOpener-4f7ceee8cd66bf2f0907e201f9289fa0-1] regionserver.HStore(310): Store=4f7ceee8cd66bf2f0907e201f9289fa0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:12,848 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:12,849 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:12,853 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:56:12,856 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:56:12,857 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 4f7ceee8cd66bf2f0907e201f9289fa0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74191069, jitterRate=0.10553307831287384}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-30 03:56:12,857 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 4f7ceee8cd66bf2f0907e201f9289fa0: 2023-03-30 03:56:12,860 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0., pid=6, masterSystemTime=1680148572827 2023-03-30 03:56:12,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:56:12,863 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:56:12,865 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4f7ceee8cd66bf2f0907e201f9289fa0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:12,865 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680148572864"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148572864"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148572864"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148572864"}]},"ts":"1680148572864"} 2023-03-30 03:56:12,871 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-30 03:56:12,871 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 4f7ceee8cd66bf2f0907e201f9289fa0, server=jenkins-hbase20.apache.org,41157,1680148568851 in 196 msec 2023-03-30 03:56:12,874 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-30 03:56:12,874 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=4f7ceee8cd66bf2f0907e201f9289fa0, ASSIGN in 366 msec 2023-03-30 03:56:12,875 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 03:56:12,876 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148572876"}]},"ts":"1680148572876"} 2023-03-30 03:56:12,880 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-30 03:56:12,883 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 03:56:12,886 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3400 sec 2023-03-30 03:56:12,959 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-30 03:56:12,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-30 03:56:12,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:56:13,003 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-30 03:56:13,022 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-30 03:56:13,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-03-30 03:56:13,037 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-30 03:56:13,047 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-30 03:56:13,052 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-03-30 03:56:13,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-30 03:56:13,064 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-30 03:56:13,064 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.135sec 2023-03-30 03:56:13,066 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-30 03:56:13,068 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-30 03:56:13,068 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-30 03:56:13,070 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37051,1680148567795-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-30 03:56:13,070 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37051,1680148567795-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-30 03:56:13,083 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-30 03:56:13,099 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x6dc6e9d6 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:13,105 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ddc537c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:13,118 DEBUG [hconnection-0x68a18665-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:13,135 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37594, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:13,143 INFO [Listener at localhost.localdomain/37215] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:56:13,159 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=171, ProcessCount=166, AvailableMemoryMB=10940 2023-03-30 03:56:13,172 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 03:56:13,175 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34288, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 03:56:13,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:56:13,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:56:13,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:13,192 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 03:56:13,195 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 03:56:13,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-30 03:56:13,197 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,199 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f empty. 2023-03-30 03:56:13,200 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,200 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:56:13,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 03:56:13,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 03:56:13,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 03:56:13,617 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:56:13,624 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3ecc6181b1966d72d93cc5e543ef979f, NAME => 'TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp 2023-03-30 03:56:13,639 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:13,639 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3ecc6181b1966d72d93cc5e543ef979f, disabling compactions & flushes 2023-03-30 03:56:13,639 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:13,639 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:13,639 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. after waiting 0 ms 2023-03-30 03:56:13,639 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:13,639 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:13,639 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:13,644 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 03:56:13,646 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148573645"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148573645"}]},"ts":"1680148573645"} 2023-03-30 03:56:13,649 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 03:56:13,650 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 03:56:13,651 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148573651"}]},"ts":"1680148573651"} 2023-03-30 03:56:13,653 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 03:56:13,657 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ecc6181b1966d72d93cc5e543ef979f, ASSIGN}] 2023-03-30 03:56:13,660 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ecc6181b1966d72d93cc5e543ef979f, ASSIGN 2023-03-30 03:56:13,662 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ecc6181b1966d72d93cc5e543ef979f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=false 2023-03-30 03:56:13,813 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3ecc6181b1966d72d93cc5e543ef979f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:13,814 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148573813"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148573813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148573813"}]},"ts":"1680148573813"} 2023-03-30 03:56:13,817 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:56:13,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 03:56:13,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:13,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3ecc6181b1966d72d93cc5e543ef979f, NAME => 'TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:56:13,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:13,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,980 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,981 DEBUG [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A 2023-03-30 03:56:13,982 DEBUG [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A 2023-03-30 03:56:13,983 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:13,983 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ecc6181b1966d72d93cc5e543ef979f columnFamilyName A 2023-03-30 03:56:13,984 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.HStore(310): Store=3ecc6181b1966d72d93cc5e543ef979f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:13,984 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,986 DEBUG [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B 2023-03-30 03:56:13,986 DEBUG [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B 2023-03-30 03:56:13,987 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:13,987 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ecc6181b1966d72d93cc5e543ef979f columnFamilyName B 2023-03-30 03:56:13,988 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.HStore(310): Store=3ecc6181b1966d72d93cc5e543ef979f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:13,988 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,990 DEBUG [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C 2023-03-30 03:56:13,990 DEBUG [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C 2023-03-30 03:56:13,991 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:13,991 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ecc6181b1966d72d93cc5e543ef979f columnFamilyName C 2023-03-30 03:56:13,991 INFO [StoreOpener-3ecc6181b1966d72d93cc5e543ef979f-1] regionserver.HStore(310): Store=3ecc6181b1966d72d93cc5e543ef979f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:13,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:13,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:13,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:56:14,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:14,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:56:14,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 3ecc6181b1966d72d93cc5e543ef979f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65631705, jitterRate=-0.02201138436794281}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:56:14,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:14,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., pid=11, masterSystemTime=1680148573971 2023-03-30 03:56:14,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:14,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:14,012 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3ecc6181b1966d72d93cc5e543ef979f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,012 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148574012"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148574012"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148574012"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148574012"}]},"ts":"1680148574012"} 2023-03-30 03:56:14,018 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-30 03:56:14,018 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 in 198 msec 2023-03-30 03:56:14,021 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-30 03:56:14,022 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ecc6181b1966d72d93cc5e543ef979f, ASSIGN in 361 msec 2023-03-30 03:56:14,023 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 03:56:14,023 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148574023"}]},"ts":"1680148574023"} 2023-03-30 03:56:14,025 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 03:56:14,028 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 03:56:14,030 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 840 msec 2023-03-30 03:56:14,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 03:56:14,324 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-30 03:56:14,328 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x458f9a9e to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,332 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d3fb5c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,335 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,339 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37606, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,343 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 03:56:14,346 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34302, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 03:56:14,355 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x20e4e4f3 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,359 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5610151a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,360 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x142c8aea to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,364 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45864842, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,365 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x6362e5a0 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,370 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3167991b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,370 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x1f537ba4 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,374 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ca212c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,374 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x046b5e40 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,385 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29be3c5c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,386 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x5fe050a3 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,393 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e3f14f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,394 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x655398b8 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,425 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38ebb9f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,426 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x13112432 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,430 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76d073d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,430 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x776686ab to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:14,434 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dcc1cc1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:14,462 DEBUG [hconnection-0x7328ed9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,462 DEBUG [hconnection-0x4f512920-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,464 DEBUG [hconnection-0x217990b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,465 DEBUG [hconnection-0x50bd83ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,465 DEBUG [hconnection-0xac8a7e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,465 DEBUG [hconnection-0x65feb4b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,467 DEBUG [hconnection-0x33f7e9c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,472 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37626, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,472 DEBUG [hconnection-0x2da18d42-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,472 DEBUG [hconnection-0x6ff5cb68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:14,475 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37610, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:14,483 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37638, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,484 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37658, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,485 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37688, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,492 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37654, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,492 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37686, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,500 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37670, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,501 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37704, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:14,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:14,519 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:14,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:14,520 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:14,520 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:14,521 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,521 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,522 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:14,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:14,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:14,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:14,548 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:14,548 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:14,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:14,572 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:14,572 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:14,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:14,576 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:14,578 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:14,578 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:14,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 03:56:14,579 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:14,585 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,585 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:14,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:14,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@203d71b5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:14,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:14,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:14,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:14,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:14,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:14,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:14,606 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:14,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:14,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:14,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:14,608 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:14,609 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:14,609 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:14,609 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:14,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:14,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:14,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:14,620 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:14,620 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:14,620 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:14,620 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:14,620 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:14,633 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,633 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,633 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:14,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,660 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,660 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,660 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,660 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:14,661 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,661 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:14,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:14,661 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:14,661 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,661 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,661 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,661 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:14,664 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:14,664 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:14,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148634701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148634700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148634723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148634723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148634727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,764 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:14,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:14,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:14,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:14,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:14,802 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:14,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:14,806 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:14,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:14,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,808 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:14,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,809 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,810 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,811 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:14,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:14,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:14,812 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:14,812 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:14,813 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:14,813 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:14,813 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:14,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:14,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:14,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:14,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:14,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:14,820 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,820 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d8b799[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:14,822 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:14,822 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,823 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,823 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:14,823 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:14,823 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:14,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:14,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,825 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:14,825 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:14,825 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:14,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:14,830 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:14,831 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:14,831 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:14,831 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:14,831 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:14,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:14,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:14,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,839 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:14,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:14,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,848 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:14,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:14,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:14,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:14,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,849 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,849 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:14,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:14,849 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:14,849 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:14,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:14,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148634859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148634859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148634860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148634860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:14,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148634864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,950 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:14,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:14,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:14,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:14,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:14,964 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:14,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:14,965 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:14,965 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:14,965 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,967 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:14,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:14,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:14,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:14,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:14,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:14,972 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:14,972 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:14,974 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:14,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:14,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:14,978 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:14,978 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:14,978 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:14,978 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:14,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:14,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:14,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:14,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:14,983 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:14,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:14,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:14,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0505bc[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:14,987 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:14,987 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,988 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:14,988 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:14,988 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:14,988 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:14,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:14,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:14,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:14,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:14,991 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:14,996 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:14,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:14,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:14,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:14,996 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:15,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:15,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:15,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:15,016 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:15,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,017 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,017 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,016 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:15,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148635064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148635066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148635064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148635067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148635069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,119 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:15,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:15,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:15,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:15,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:15,129 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:15,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:15,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:15,131 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,131 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,132 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:15,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:15,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:15,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:15,136 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:15,136 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,136 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:15,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:15,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:15,138 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,138 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:15,138 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:15,138 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:15,139 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:15,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:15,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,146 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,146 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,146 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:15,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68c9fc33[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:15,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:15,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,150 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/11eb7f8c1ee340d3a93a94034aa7b93d 2023-03-30 03:56:15,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,152 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:15,152 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:15,152 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:15,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,160 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:15,160 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,160 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:15,160 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:15,161 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:15,160 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:15,161 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:15,161 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:15,167 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,167 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,168 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,172 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:15,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:15,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:15,179 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,180 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:15,181 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:15,281 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:15,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:15,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:15,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:15,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:15,288 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:15,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:15,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:15,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,292 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,293 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,293 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:15,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:15,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:15,295 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:15,295 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,296 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:15,298 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,298 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:15,298 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:15,298 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,299 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:15,299 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:15,299 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,299 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:15,299 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:15,302 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:15,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b29846[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:15,306 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:15,306 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,307 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:15,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:15,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:15,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:15,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:15,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:15,309 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:15,313 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:15,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:15,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:15,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:15,314 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:15,318 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,318 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,318 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,320 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,322 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:15,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:15,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:15,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,327 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:15,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,327 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148635371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148635371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148635371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148635372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148635375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,428 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:15,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:15,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:15,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:15,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:15,436 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:15,436 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,442 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:15,443 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:15,443 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,444 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,445 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,445 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,446 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,446 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:15,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:15,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:15,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:15,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:15,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:15,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:15,451 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:15,451 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:15,451 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,451 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:15,451 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:15,451 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,452 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:15,452 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:15,452 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,452 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:15,452 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:15,468 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@641197d6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:15,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:15,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,473 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,474 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:15,474 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:15,474 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:15,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,475 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:15,477 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:15,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:15,483 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:15,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:15,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:15,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:15,484 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:15,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,487 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:15,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:15,501 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,502 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:15,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:15,502 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:15,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,603 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:15,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:15,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:15,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:15,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:15,608 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:15,608 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,609 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:15,609 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:15,609 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,609 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,610 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:15,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:15,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:15,612 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:15,612 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:15,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:15,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:15,613 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,613 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:15,613 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:15,614 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:15,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:15,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:15,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:15,617 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,618 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,618 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:15,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530e560d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:15,620 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:15,620 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,621 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:15,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:15,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:15,621 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,621 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:15,622 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:15,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,622 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,622 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:15,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:15,626 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:15,626 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:15,626 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:15,626 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:15,626 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:15,634 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,634 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,634 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:15,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:15,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,646 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:15,647 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:15,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:15,647 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,647 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,647 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,647 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/fc1ff4cba8ae4f3f96aaa20a9f3b4cf3 2023-03-30 03:56:15,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/2ec8035aacc44ba190936bdf3806945f 2023-03-30 03:56:15,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/11eb7f8c1ee340d3a93a94034aa7b93d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/11eb7f8c1ee340d3a93a94034aa7b93d 2023-03-30 03:56:15,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/11eb7f8c1ee340d3a93a94034aa7b93d, entries=150, sequenceid=15, filesize=11.6 K 2023-03-30 03:56:15,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/fc1ff4cba8ae4f3f96aaa20a9f3b4cf3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fc1ff4cba8ae4f3f96aaa20a9f3b4cf3 2023-03-30 03:56:15,747 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:15,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:15,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:15,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:15,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:15,757 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:15,757 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fc1ff4cba8ae4f3f96aaa20a9f3b4cf3, entries=150, sequenceid=15, filesize=11.6 K 2023-03-30 03:56:15,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:15,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:15,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,768 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/2ec8035aacc44ba190936bdf3806945f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2ec8035aacc44ba190936bdf3806945f 2023-03-30 03:56:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:15,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:15,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:15,780 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:15,780 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,781 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:15,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2ec8035aacc44ba190936bdf3806945f, entries=150, sequenceid=15, filesize=11.6 K 2023-03-30 03:56:15,785 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:15,786 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,786 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:15,786 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:15,787 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:15,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 3ecc6181b1966d72d93cc5e543ef979f in 1209ms, sequenceid=15, compaction requested=false 2023-03-30 03:56:15,791 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:15,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:15,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:15,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:15,794 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-30 03:56:15,794 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,794 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,794 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,796 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:15,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:15,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7401103b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:15,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:15,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,798 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,799 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:15,799 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:15,799 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:15,801 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:15,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:15,802 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:15,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:15,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:15,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:15,802 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:15,804 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,804 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,810 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,821 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,821 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,822 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:15,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:15,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,825 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:15,825 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:15,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:15,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-03-30 03:56:15,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:15,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:15,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:15,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:15,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:15,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:15,926 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:15,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:15,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:15,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:15,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148635926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:15,951 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:15,952 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:15,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:15,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,956 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:15,956 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:15,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:15,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:15,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:15,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:15,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:15,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:15,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:15,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:15,965 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,965 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:15,965 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:15,965 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:15,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:15,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:15,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:15,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:15,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148635939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148635941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:15,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148635952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148635944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,976 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,976 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,977 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,977 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4346c655[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:15,979 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:15,979 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,980 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:15,980 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:15,980 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:15,981 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:15,981 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,981 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:15,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:15,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:15,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:15,991 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:15,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:15,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:15,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:15,991 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:15,992 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,993 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:15,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,993 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:15,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:15,994 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:16,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:16,003 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:16,003 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:16,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148636055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148636073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148636073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148636073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148636075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,103 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:16,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:16,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:16,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:16,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:16,112 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:16,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:16,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:16,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,115 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:16,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:16,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:16,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:16,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,120 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:16,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:16,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:16,123 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,123 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:16,130 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:16,130 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:16,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:16,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:16,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:16,133 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:16,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0146cc[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:16,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:16,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,137 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,138 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,138 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:16,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:16,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:16,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:16,139 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,140 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:16,140 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,140 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:16,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:16,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:16,147 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:16,147 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:16,147 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:16,147 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:16,147 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:16,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,153 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,155 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,163 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,163 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,164 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,164 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,164 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,164 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,164 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,165 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:16,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:16,165 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,165 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,165 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:16,165 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:16,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:16,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148636261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,265 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:16,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:16,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:16,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:16,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:16,271 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:16,271 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,272 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:16,272 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:16,273 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,273 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,274 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,274 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,274 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:16,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:16,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:16,276 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:16,276 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:16,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148636280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,282 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:16,282 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:16,282 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,282 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:16,283 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:16,283 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:16,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:16,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:16,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:16,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148636283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,285 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,285 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148636284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,286 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,286 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148636284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:16,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9ed39f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:16,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:16,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:16,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:16,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:16,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:16,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:16,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:16,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:16,299 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:16,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:16,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:16,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:16,299 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:16,302 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,302 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,305 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,310 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:16,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:16,311 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:16,311 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:16,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:16,365 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/d13d757ae1444c1bb3e79aa76a660ed4 2023-03-30 03:56:16,411 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:16,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:16,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6ebb0a72bad34ebcbffb9b6da4018922 2023-03-30 03:56:16,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:16,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:16,428 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:16,428 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,429 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:16,429 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:16,430 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,430 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,432 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,435 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,435 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:16,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:16,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:16,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:16,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:16,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:16,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:16,439 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,439 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:16,439 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:16,439 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:16,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:16,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:16,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:16,443 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,443 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,444 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,444 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:16,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76628a3f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:16,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:16,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:16,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:16,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:16,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:16,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,486 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:16,486 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,486 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:16,491 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:16,491 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:16,491 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:16,491 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:16,491 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:16,493 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,494 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,494 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,496 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,501 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:16,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:16,503 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,503 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,503 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:16,503 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:16,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/7ca599eddef744bc80ec474584abe48b 2023-03-30 03:56:16,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/d13d757ae1444c1bb3e79aa76a660ed4 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d13d757ae1444c1bb3e79aa76a660ed4 2023-03-30 03:56:16,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d13d757ae1444c1bb3e79aa76a660ed4, entries=150, sequenceid=37, filesize=11.6 K 2023-03-30 03:56:16,539 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6ebb0a72bad34ebcbffb9b6da4018922 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ebb0a72bad34ebcbffb9b6da4018922 2023-03-30 03:56:16,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ebb0a72bad34ebcbffb9b6da4018922, entries=150, sequenceid=37, filesize=11.6 K 2023-03-30 03:56:16,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/7ca599eddef744bc80ec474584abe48b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7ca599eddef744bc80ec474584abe48b 2023-03-30 03:56:16,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148636569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7ca599eddef744bc80ec474584abe48b, entries=150, sequenceid=37, filesize=11.6 K 2023-03-30 03:56:16,584 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=46.96 KB/48090 for 3ecc6181b1966d72d93cc5e543ef979f in 678ms, sequenceid=37, compaction requested=false 2023-03-30 03:56:16,584 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:16,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:16,594 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:56:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:16,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:16,604 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:16,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:16,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:16,632 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:16,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:16,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:16,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,646 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:16,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:16,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:16,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:16,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:16,657 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:16,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:16,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:16,658 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,658 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:16,658 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:16,658 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:16,661 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,663 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,663 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,663 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:16,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3638ec8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:16,665 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:16,666 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,666 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,667 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:16,667 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:16,667 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:16,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:16,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:16,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:16,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:16,675 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:16,675 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:16,675 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:16,675 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:16,675 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:16,684 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,684 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,684 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148636673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,685 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148636674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148636678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148636681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:16,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:16,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,694 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:16,694 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:16,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,695 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:16,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148636787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148636788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,796 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:16,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:16,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:16,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148636795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148636795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:16,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:16,803 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:16,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,806 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:16,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:16,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,808 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,808 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,809 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,809 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:16,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:16,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:16,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:16,811 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:16,811 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,812 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:16,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:16,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:16,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:16,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:16,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:16,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:16,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:16,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:16,818 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:16,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f58f4e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:16,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:16,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,822 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,822 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:16,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:16,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:16,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:16,823 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,823 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,824 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,824 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:16,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:16,827 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:16,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:16,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:16,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:16,827 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:16,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,830 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:16,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:16,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,837 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:16,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:16,886 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 03:56:16,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:16,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:16,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:16,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:16,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:16,945 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:16,947 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,952 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:16,952 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:16,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,954 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:16,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:16,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:16,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:16,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:16,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:16,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:16,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:16,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:16,964 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,964 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:16,964 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:16,964 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:16,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:16,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:16,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:16,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:16,968 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,968 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,968 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,968 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:16,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e8ba2a4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:16,971 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:16,971 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,976 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:16,977 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:16,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:16,977 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:16,977 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:16,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:16,978 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:16,984 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:16,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:16,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:16,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:16,984 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:16,993 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,993 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:16,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,993 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-30 03:56:16,993 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:16,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:16,994 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148636993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:16,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148636993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:16,997 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-30 03:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:16,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:16,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:16,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:16,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:16,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:16,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:16,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:17,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,002 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:17,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:17,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148637003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148637004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/7912677c614e46bcbf22177bde1991e9 2023-03-30 03:56:17,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148637081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,102 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:17,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:17,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/ce8e80dba5bd4e35979baff6dcf97c18 2023-03-30 03:56:17,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:17,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:17,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:17,114 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:17,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:17,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:17,119 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,119 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,122 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,122 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:17,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:17,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:17,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:17,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:17,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:17,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:17,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:17,125 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,125 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:17,126 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:17,126 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:17,136 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c5ae838[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:17,139 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:17,139 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:17,143 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,143 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:17,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:17,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:17,152 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:17,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,152 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,153 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/3ca54b7bb9a04fb8908a7bbab2ff5ae4 2023-03-30 03:56:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:17,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:17,165 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:17,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:17,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:17,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:17,166 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:17,189 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,189 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,190 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,190 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,194 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:17,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:17,195 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:17,195 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:17,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/7912677c614e46bcbf22177bde1991e9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7912677c614e46bcbf22177bde1991e9 2023-03-30 03:56:17,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7912677c614e46bcbf22177bde1991e9, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 03:56:17,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/ce8e80dba5bd4e35979baff6dcf97c18 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ce8e80dba5bd4e35979baff6dcf97c18 2023-03-30 03:56:17,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ce8e80dba5bd4e35979baff6dcf97c18, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 03:56:17,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/3ca54b7bb9a04fb8908a7bbab2ff5ae4 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3ca54b7bb9a04fb8908a7bbab2ff5ae4 2023-03-30 03:56:17,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3ca54b7bb9a04fb8908a7bbab2ff5ae4, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 03:56:17,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3ecc6181b1966d72d93cc5e543ef979f in 647ms, sequenceid=49, compaction requested=true 2023-03-30 03:56:17,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:17,246 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:17,246 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:17,246 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:17,246 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:17,246 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:17,251 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:17,254 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:17,254 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,254 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fc1ff4cba8ae4f3f96aaa20a9f3b4cf3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ebb0a72bad34ebcbffb9b6da4018922, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ce8e80dba5bd4e35979baff6dcf97c18] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=34.8 K 2023-03-30 03:56:17,255 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:17,255 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:17,256 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting fc1ff4cba8ae4f3f96aaa20a9f3b4cf3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680148574539 2023-03-30 03:56:17,259 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6ebb0a72bad34ebcbffb9b6da4018922, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680148574708 2023-03-30 03:56:17,260 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,260 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/11eb7f8c1ee340d3a93a94034aa7b93d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d13d757ae1444c1bb3e79aa76a660ed4, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7912677c614e46bcbf22177bde1991e9] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=34.8 K 2023-03-30 03:56:17,260 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ce8e80dba5bd4e35979baff6dcf97c18, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148575933 2023-03-30 03:56:17,261 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 11eb7f8c1ee340d3a93a94034aa7b93d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680148574539 2023-03-30 03:56:17,263 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting d13d757ae1444c1bb3e79aa76a660ed4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680148574708 2023-03-30 03:56:17,264 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 7912677c614e46bcbf22177bde1991e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148575933 2023-03-30 03:56:17,288 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#9 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:17,299 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:17,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:17,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:17,304 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#10 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:17,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:17,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:17,312 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:17,312 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:17,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:17,316 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,316 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,320 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,323 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,323 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:17,333 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-03-30 03:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:17,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:17,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148576661, max timestamp=1680148577320Num uniques 859; 2023-03-30 03:56:17,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:17,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:17,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:17,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:17,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:17,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:17,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:17,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:17,349 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:17,349 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:17,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:17,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:17,351 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,351 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:17,351 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:17,351 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,352 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:17,352 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:17,352 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,353 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:17,353 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:17,355 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,355 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,355 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,355 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,372 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/8f0ee0b778d64f85ba9c3772f6102b83 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8f0ee0b778d64f85ba9c3772f6102b83 2023-03-30 03:56:17,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:17,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d024b5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:17,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:17,374 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,375 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,376 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,376 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:17,376 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:17,376 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:17,376 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:17,376 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/264772cfba1c4ec6b7ae7bbc2ebcc33a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/264772cfba1c4ec6b7ae7bbc2ebcc33a 2023-03-30 03:56:17,378 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:17,378 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,378 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:17,380 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148637379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148637379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148637380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148637378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/64dd662831be4b81b94c970510615601 2023-03-30 03:56:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,408 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into 8f0ee0b778d64f85ba9c3772f6102b83(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:17,408 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into 264772cfba1c4ec6b7ae7bbc2ebcc33a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:17,408 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:17,408 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:17,409 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148577245; duration=0sec 2023-03-30 03:56:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,409 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148577246; duration=0sec 2023-03-30 03:56:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,410 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:17,410 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:17,411 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:17,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:17,412 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:17,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:17,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:17,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:17,412 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:17,412 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:17,413 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:17,413 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,413 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2ec8035aacc44ba190936bdf3806945f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7ca599eddef744bc80ec474584abe48b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3ca54b7bb9a04fb8908a7bbab2ff5ae4] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=34.8 K 2023-03-30 03:56:17,414 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,414 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,414 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,414 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2ec8035aacc44ba190936bdf3806945f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680148574539 2023-03-30 03:56:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,422 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 7ca599eddef744bc80ec474584abe48b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680148574708 2023-03-30 03:56:17,422 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,424 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3ca54b7bb9a04fb8908a7bbab2ff5ae4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148575933 2023-03-30 03:56:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,439 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,440 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:17,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:17,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:17,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:17,450 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:17,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/144689632f964c49b5c149c91c3506f6 2023-03-30 03:56:17,453 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#13 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:17,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148637503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148637503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148637503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148637508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/d7cb3efc92ab4c48b1bfb28354c381c7 2023-03-30 03:56:17,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/64dd662831be4b81b94c970510615601 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/64dd662831be4b81b94c970510615601 2023-03-30 03:56:17,530 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/873a1330f82146fa80e5a0339702d110 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/873a1330f82146fa80e5a0339702d110 2023-03-30 03:56:17,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/64dd662831be4b81b94c970510615601, entries=200, sequenceid=72, filesize=13.9 K 2023-03-30 03:56:17,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/144689632f964c49b5c149c91c3506f6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/144689632f964c49b5c149c91c3506f6 2023-03-30 03:56:17,550 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:17,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:17,554 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into 873a1330f82146fa80e5a0339702d110(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:17,555 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:17,555 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148577246; duration=0sec 2023-03-30 03:56:17,555 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:17,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:17,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:17,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:17,559 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:17,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,560 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:17,560 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:17,561 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,561 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,563 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,563 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,565 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:17,565 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:17,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:17,566 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:17,566 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,571 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:17,571 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:17,571 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:17,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/144689632f964c49b5c149c91c3506f6, entries=150, sequenceid=72, filesize=11.6 K 2023-03-30 03:56:17,573 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,573 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:17,574 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:17,574 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,574 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:17,574 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:17,574 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,574 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:17,574 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:17,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/d7cb3efc92ab4c48b1bfb28354c381c7 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d7cb3efc92ab4c48b1bfb28354c381c7 2023-03-30 03:56:17,579 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,579 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dd1507e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:17,581 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:17,581 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,584 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,584 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:17,585 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:17,585 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:17,585 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,585 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:17,587 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:17,587 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,587 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:17,587 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d7cb3efc92ab4c48b1bfb28354c381c7, entries=150, sequenceid=72, filesize=11.6 K 2023-03-30 03:56:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:17,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~102.87 KB/105340, heapSize ~301.91 KB/309152, currentSize=80.51 KB/82440 for 3ecc6181b1966d72d93cc5e543ef979f in 262ms, sequenceid=72, compaction requested=false 2023-03-30 03:56:17,593 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:17,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:17,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:17,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:17,594 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:17,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:17,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:17,601 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,601 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,601 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,606 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,609 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,609 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,609 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,609 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,609 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,609 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:17,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:17,615 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:17,615 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:17,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:17,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:17,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-30 03:56:17,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:17,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:17,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:17,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:17,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:17,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:17,715 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:17,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:17,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:17,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:17,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:17,726 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:17,726 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:17,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:17,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,731 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,732 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,737 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,737 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:17,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:17,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:17,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,741 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:17,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:17,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:17,742 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,742 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:17,742 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:17,742 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:17,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:17,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:17,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:17,746 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,746 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/98e140b9a99f4754a7ade4b806d97d23 2023-03-30 03:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e4c59e2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:17,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:17,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:17,764 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,765 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:17,765 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:17,765 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:17,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:17,766 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,767 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148637766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148637766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148637769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148637770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:17,781 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:17,781 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:17,781 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:17,781 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:17,781 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:17,784 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,784 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,784 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,785 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,794 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:17,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:17,795 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,795 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,795 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:17,795 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:17,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/c81011d549254002a2cec535c43469b6 2023-03-30 03:56:17,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148637878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148637878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148637879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:17,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148637879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,895 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:17,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:17,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:17,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:17,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:17,900 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:17,900 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:17,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:17,902 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,902 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,903 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,904 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,904 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:17,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:17,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:17,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:17,910 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:17,910 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:17,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:17,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:17,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:17,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:17,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:17,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:17,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:17,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:17,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:17,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:17,918 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,918 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,919 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,919 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:17,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453c7f3b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:17,922 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:17,922 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,924 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:17,924 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:17,924 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:17,925 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,925 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:17,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,925 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,925 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,925 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:17,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:17,929 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:17,929 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:17,929 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:17,929 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:17,929 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:17,931 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,931 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,931 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:17,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,939 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-30 03:56:17,940 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-30 03:56:17,943 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:17,943 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:17,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:17,943 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:17,943 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:17,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:17,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:17,943 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:17,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:17,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:17,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:17,948 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:17,948 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:17,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,048 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:18,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:18,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:18,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:18,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:18,052 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:18,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:18,053 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:18,053 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:18,053 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,053 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,054 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:18,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:18,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:18,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:18,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:18,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:18,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:18,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:18,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:18,061 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:18,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:18,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:18,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:18,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:18,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:18,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:18,067 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,067 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@676b1ea5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:18,069 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:18,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,071 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:18,071 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:18,071 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:18,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:18,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,072 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,072 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:18,072 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:18,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:18,075 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:18,075 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:18,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:18,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:18,076 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:18,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148638081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:18,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:18,085 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148638083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148638084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,086 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148638085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148638089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:18,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:18,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:18,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:18,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:18,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,094 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:18,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:18,094 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:18,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,094 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:18,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:18,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:18,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,194 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:18,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:18,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:18,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:18,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:18,199 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:18,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:18,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:18,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:18,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,201 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:18,201 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:18,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:18,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:18,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:18,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:18,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:18,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:18,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:18,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:18,203 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,203 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:18,204 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:18,204 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:18,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:18,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:18,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:18,207 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,207 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,207 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,207 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@401ee096[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:18,210 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:18,210 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,212 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,212 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:18,212 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,212 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:18,212 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:18,212 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:18,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:18,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:18,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:18,214 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:18,218 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:18,218 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:18,218 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:18,218 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:18,218 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:18,234 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,234 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:18,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,236 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,237 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,239 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:18,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:18,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:18,243 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:18,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:18,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:18,244 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:18,244 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:18,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:18,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:18,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/d34cf8cf2c0a43a4b464d9222316e3ba 2023-03-30 03:56:18,260 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/98e140b9a99f4754a7ade4b806d97d23 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/98e140b9a99f4754a7ade4b806d97d23 2023-03-30 03:56:18,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/98e140b9a99f4754a7ade4b806d97d23, entries=200, sequenceid=91, filesize=13.9 K 2023-03-30 03:56:18,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/c81011d549254002a2cec535c43469b6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/c81011d549254002a2cec535c43469b6 2023-03-30 03:56:18,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/c81011d549254002a2cec535c43469b6, entries=150, sequenceid=91, filesize=11.6 K 2023-03-30 03:56:18,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/d34cf8cf2c0a43a4b464d9222316e3ba as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d34cf8cf2c0a43a4b464d9222316e3ba 2023-03-30 03:56:18,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d34cf8cf2c0a43a4b464d9222316e3ba, entries=150, sequenceid=91, filesize=11.6 K 2023-03-30 03:56:18,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 3ecc6181b1966d72d93cc5e543ef979f in 614ms, sequenceid=91, compaction requested=true 2023-03-30 03:56:18,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:18,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:18,328 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:18,328 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:18,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:18,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:18,330 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:18,330 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:18,330 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,330 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8f0ee0b778d64f85ba9c3772f6102b83, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/144689632f964c49b5c149c91c3506f6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/c81011d549254002a2cec535c43469b6] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=34.9 K 2023-03-30 03:56:18,331 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8f0ee0b778d64f85ba9c3772f6102b83, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148575933 2023-03-30 03:56:18,331 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40441 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:18,332 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:18,332 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,332 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/264772cfba1c4ec6b7ae7bbc2ebcc33a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/64dd662831be4b81b94c970510615601, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/98e140b9a99f4754a7ade4b806d97d23] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=39.5 K 2023-03-30 03:56:18,333 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 144689632f964c49b5c149c91c3506f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680148576675 2023-03-30 03:56:18,333 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 264772cfba1c4ec6b7ae7bbc2ebcc33a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148575933 2023-03-30 03:56:18,334 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c81011d549254002a2cec535c43469b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1680148577369 2023-03-30 03:56:18,336 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 64dd662831be4b81b94c970510615601, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680148576675 2023-03-30 03:56:18,337 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 98e140b9a99f4754a7ade4b806d97d23, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1680148577369 2023-03-30 03:56:18,344 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:18,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:18,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:18,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:18,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:18,362 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:18,362 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:18,362 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:18,362 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:18,363 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,363 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,364 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:18,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:18,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:18,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:18,365 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:18,369 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#19 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:18,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:18,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:18,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:18,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:18,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:18,386 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:18,386 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:18,388 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,388 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:18,389 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-30 03:56:18,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:18,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:18,400 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:18,400 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:18,400 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:18,400 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:18,400 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:18,400 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:18,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148638440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148638440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148638441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148638447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148638554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148638555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148638556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148638557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148638758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148638759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148638759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:18,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148638759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,837 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/2f1835b3650445a99f4a392b589def42 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2f1835b3650445a99f4a392b589def42 2023-03-30 03:56:18,840 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/d3cc4b826f34467d9e0a8437980a5039 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d3cc4b826f34467d9e0a8437980a5039 2023-03-30 03:56:18,851 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into 2f1835b3650445a99f4a392b589def42(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:18,852 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into d3cc4b826f34467d9e0a8437980a5039(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:18,853 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:18,853 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:18,853 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148578328; duration=0sec 2023-03-30 03:56:18,853 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148578328; duration=0sec 2023-03-30 03:56:18,853 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:18,853 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:18,853 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:18,858 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:18,858 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:18,858 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/bf7bc7b58f5040a79cd75a571b169afd 2023-03-30 03:56:18,858 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,858 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/873a1330f82146fa80e5a0339702d110, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d7cb3efc92ab4c48b1bfb28354c381c7, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d34cf8cf2c0a43a4b464d9222316e3ba] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=34.9 K 2023-03-30 03:56:18,860 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 873a1330f82146fa80e5a0339702d110, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148575933 2023-03-30 03:56:18,863 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d7cb3efc92ab4c48b1bfb28354c381c7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680148576675 2023-03-30 03:56:18,864 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d34cf8cf2c0a43a4b464d9222316e3ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1680148577369 2023-03-30 03:56:18,883 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#22 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:18,890 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/b7b84a34fa774a65b54544c2833cb87b 2023-03-30 03:56:18,925 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/8dd3137f3a0b437684fd4e1fb8bb1e6a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/8dd3137f3a0b437684fd4e1fb8bb1e6a 2023-03-30 03:56:18,925 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/aeffcd57619d4a4da0ede7f5d35a2831 2023-03-30 03:56:18,934 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into 8dd3137f3a0b437684fd4e1fb8bb1e6a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:18,939 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:18,939 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148578328; duration=0sec 2023-03-30 03:56:18,939 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/bf7bc7b58f5040a79cd75a571b169afd as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/bf7bc7b58f5040a79cd75a571b169afd 2023-03-30 03:56:18,939 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:18,948 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/bf7bc7b58f5040a79cd75a571b169afd, entries=150, sequenceid=109, filesize=11.6 K 2023-03-30 03:56:18,951 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/b7b84a34fa774a65b54544c2833cb87b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/b7b84a34fa774a65b54544c2833cb87b 2023-03-30 03:56:18,963 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/b7b84a34fa774a65b54544c2833cb87b, entries=150, sequenceid=109, filesize=11.6 K 2023-03-30 03:56:18,968 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/aeffcd57619d4a4da0ede7f5d35a2831 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/aeffcd57619d4a4da0ede7f5d35a2831 2023-03-30 03:56:18,977 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/aeffcd57619d4a4da0ede7f5d35a2831, entries=150, sequenceid=109, filesize=11.6 K 2023-03-30 03:56:18,979 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 3ecc6181b1966d72d93cc5e543ef979f in 590ms, sequenceid=109, compaction requested=false 2023-03-30 03:56:18,979 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:18,979 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:18,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:18,980 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:18,980 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,980 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:18,980 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:18,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c51223e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:18,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:18,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:18,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:18,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:18,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:18,985 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,985 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:18,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:18,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:18,986 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:18,989 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:18,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:18,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:18,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:18,989 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:18,990 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,990 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,990 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:18,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:18,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:19,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:19,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:19,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:19,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:19,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:19,002 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:19,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:19,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,001 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:19,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:19,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:19,065 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-30 03:56:19,066 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:19,066 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:19,066 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:19,066 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:19,066 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:19,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:19,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148639084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148639087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148639088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148639092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,102 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:19,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:19,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/68e8f009e41a42d99bb0440f5d526df6 2023-03-30 03:56:19,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:19,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:19,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:19,120 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:19,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:19,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:19,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:19,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,122 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:19,122 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,122 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,122 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:19,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:19,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:19,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:19,142 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:19,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:19,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:19,143 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:19,144 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:19,150 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:19,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,152 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:19,153 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,153 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:19,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:19,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab25128[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:19,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:19,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:19,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:19,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:19,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:19,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:19,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:19,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:19,158 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:19,162 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:19,162 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:19,162 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:19,162 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:19,162 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:19,163 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,163 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:19,163 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:19,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:19,164 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/efe17035b6e7491aaf6462edf6f454db 2023-03-30 03:56:19,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:19,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:19,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:19,177 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:19,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:19,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:19,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,178 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:19,178 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:19,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:19,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148639190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148639195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148639195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148639197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/92556dd6cf7b4c4ca325cd9d9963ab2a 2023-03-30 03:56:19,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/68e8f009e41a42d99bb0440f5d526df6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/68e8f009e41a42d99bb0440f5d526df6 2023-03-30 03:56:19,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/68e8f009e41a42d99bb0440f5d526df6, entries=300, sequenceid=130, filesize=18.6 K 2023-03-30 03:56:19,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/efe17035b6e7491aaf6462edf6f454db as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/efe17035b6e7491aaf6462edf6f454db 2023-03-30 03:56:19,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/efe17035b6e7491aaf6462edf6f454db, entries=150, sequenceid=130, filesize=11.7 K 2023-03-30 03:56:19,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/92556dd6cf7b4c4ca325cd9d9963ab2a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/92556dd6cf7b4c4ca325cd9d9963ab2a 2023-03-30 03:56:19,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/92556dd6cf7b4c4ca325cd9d9963ab2a, entries=150, sequenceid=130, filesize=11.7 K 2023-03-30 03:56:19,278 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:19,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:19,280 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 3ecc6181b1966d72d93cc5e543ef979f in 215ms, sequenceid=130, compaction requested=true 2023-03-30 03:56:19,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:19,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:19,280 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:19,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:19,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-30 03:56:19,281 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:19,283 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:19,283 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:19,283 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:19,283 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2f1835b3650445a99f4a392b589def42, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/b7b84a34fa774a65b54544c2833cb87b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/efe17035b6e7491aaf6462edf6f454db] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=35.1 K 2023-03-30 03:56:19,283 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42984 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:19,284 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:19,284 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:19,284 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d3cc4b826f34467d9e0a8437980a5039, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/bf7bc7b58f5040a79cd75a571b169afd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/68e8f009e41a42d99bb0440f5d526df6] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=42.0 K 2023-03-30 03:56:19,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:19,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:19,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:19,287 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2f1835b3650445a99f4a392b589def42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1680148577369 2023-03-30 03:56:19,287 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:19,288 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:19,288 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:19,288 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:19,288 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b7b84a34fa774a65b54544c2833cb87b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680148577762 2023-03-30 03:56:19,288 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting d3cc4b826f34467d9e0a8437980a5039, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1680148577369 2023-03-30 03:56:19,289 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting efe17035b6e7491aaf6462edf6f454db, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680148578439 2023-03-30 03:56:19,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,289 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting bf7bc7b58f5040a79cd75a571b169afd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680148577762 2023-03-30 03:56:19,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,291 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 68e8f009e41a42d99bb0440f5d526df6, keycount=300, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680148578435 2023-03-30 03:56:19,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:19,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:19,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:19,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:19,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:19,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:19,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:19,320 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#27 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:19,334 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:19,339 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:19,342 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:19,342 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:19,342 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:19,342 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 03:56:19,343 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:19,343 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:19,344 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:19,344 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:19,344 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:19,344 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:19,386 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/13922bdaca7b4f26a45054a67b874b6c 2023-03-30 03:56:19,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:19,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:19,416 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/ba7fc0292eac42bcad19e0c382ddc397 2023-03-30 03:56:19,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148639427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148639428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148639429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148639431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148639534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148639533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148639534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148639540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148639756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148639756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148639756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:19,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148639759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,785 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/3fdf145905e449c19d5298e91e8679e0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3fdf145905e449c19d5298e91e8679e0 2023-03-30 03:56:19,794 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/5f95bdd6dc7041e0b65df8137cd18188 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5f95bdd6dc7041e0b65df8137cd18188 2023-03-30 03:56:19,803 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into 3fdf145905e449c19d5298e91e8679e0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:19,806 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into 5f95bdd6dc7041e0b65df8137cd18188(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:19,812 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:19,813 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:19,813 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148579280; duration=0sec 2023-03-30 03:56:19,813 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148579280; duration=0sec 2023-03-30 03:56:19,813 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:19,813 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:19,813 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:19,815 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:19,815 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:19,815 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:19,815 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/8dd3137f3a0b437684fd4e1fb8bb1e6a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/aeffcd57619d4a4da0ede7f5d35a2831, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/92556dd6cf7b4c4ca325cd9d9963ab2a] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=35.1 K 2023-03-30 03:56:19,816 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8dd3137f3a0b437684fd4e1fb8bb1e6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1680148577369 2023-03-30 03:56:19,817 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting aeffcd57619d4a4da0ede7f5d35a2831, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680148577762 2023-03-30 03:56:19,817 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 92556dd6cf7b4c4ca325cd9d9963ab2a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680148578439 2023-03-30 03:56:19,828 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#32 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:19,864 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/ae8670823e6342978de61f25f808441a 2023-03-30 03:56:19,874 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/13922bdaca7b4f26a45054a67b874b6c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/13922bdaca7b4f26a45054a67b874b6c 2023-03-30 03:56:19,886 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/13922bdaca7b4f26a45054a67b874b6c, entries=150, sequenceid=144, filesize=11.8 K 2023-03-30 03:56:19,892 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/ba7fc0292eac42bcad19e0c382ddc397 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ba7fc0292eac42bcad19e0c382ddc397 2023-03-30 03:56:19,900 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ba7fc0292eac42bcad19e0c382ddc397, entries=150, sequenceid=144, filesize=11.8 K 2023-03-30 03:56:19,903 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/ae8670823e6342978de61f25f808441a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ae8670823e6342978de61f25f808441a 2023-03-30 03:56:19,927 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ae8670823e6342978de61f25f808441a, entries=150, sequenceid=144, filesize=11.8 K 2023-03-30 03:56:19,929 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 3ecc6181b1966d72d93cc5e543ef979f in 587ms, sequenceid=144, compaction requested=false 2023-03-30 03:56:19,929 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:19,929 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:19,929 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:19,931 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:19,931 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,931 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:19,931 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:19,932 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,933 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:19,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:19,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:19,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:19,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1513eced[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:19,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:19,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,936 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:19,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:19,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:19,936 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,936 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:19,937 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:19,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:19,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:19,942 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:19,942 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:19,942 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:19,942 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:19,942 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:19,944 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,944 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:19,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,944 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:19,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,947 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,959 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:19,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:19,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:19,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:19,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:19,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:19,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:19,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:19,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:19,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:19,960 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:19,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:19,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:20,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:20,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:20,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 03:56:20,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:20,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:20,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:20,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:20,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:20,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:20,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:20,071 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:20,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:20,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:20,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,076 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,077 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,077 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:20,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148640089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148640091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148640092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148640093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148640102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:20,110 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:20,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:20,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:20,114 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:20,114 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:20,117 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,125 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:20,126 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:20,126 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:20,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:20,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:20,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:20,127 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:20,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22beb489[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:20,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:20,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,131 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:20,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:20,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:20,132 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,132 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:20,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:20,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:20,134 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:20,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:20,137 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:20,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:20,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:20,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:20,137 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:20,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,154 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/d5dfed8f977d4ab5adfe053b3bd79569 2023-03-30 03:56:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,164 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:20,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:20,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:20,165 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:20,165 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:20,168 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,168 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,168 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,168 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,168 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,168 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,169 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,169 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,169 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/04058b3fe2c143088e6034d52b6cfa04 2023-03-30 03:56:20,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148640198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148640199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148640203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148640203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/38139219f31d4e22b21f1467c34af3ae 2023-03-30 03:56:20,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/d5dfed8f977d4ab5adfe053b3bd79569 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d5dfed8f977d4ab5adfe053b3bd79569 2023-03-30 03:56:20,253 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/c7ba2d54bafe48f88d133401d2d580f1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c7ba2d54bafe48f88d133401d2d580f1 2023-03-30 03:56:20,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d5dfed8f977d4ab5adfe053b3bd79569, entries=150, sequenceid=165, filesize=11.8 K 2023-03-30 03:56:20,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/04058b3fe2c143088e6034d52b6cfa04 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/04058b3fe2c143088e6034d52b6cfa04 2023-03-30 03:56:20,265 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:20,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/04058b3fe2c143088e6034d52b6cfa04, entries=150, sequenceid=165, filesize=11.8 K 2023-03-30 03:56:20,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:20,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/38139219f31d4e22b21f1467c34af3ae as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/38139219f31d4e22b21f1467c34af3ae 2023-03-30 03:56:20,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:20,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:20,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:20,301 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:20,301 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,301 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:20,301 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:20,301 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,301 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,302 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,302 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,302 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,302 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into c7ba2d54bafe48f88d133401d2d580f1(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:20,302 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:20,303 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:20,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,303 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148579280; duration=0sec 2023-03-30 03:56:20,303 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:20,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:20,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:20,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/38139219f31d4e22b21f1467c34af3ae, entries=150, sequenceid=165, filesize=11.8 K 2023-03-30 03:56:20,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:20,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:20,308 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:20,308 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,308 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:20,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:20,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:20,310 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,310 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:20,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:20,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:20,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:20,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:20,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:20,312 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3ecc6181b1966d72d93cc5e543ef979f in 244ms, sequenceid=165, compaction requested=true 2023-03-30 03:56:20,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:20,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:20,313 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:20,313 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:20,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:20,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:20,315 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:20,315 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:20,315 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:20,315 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:20,315 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,315 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,315 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3fdf145905e449c19d5298e91e8679e0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ba7fc0292eac42bcad19e0c382ddc397, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/04058b3fe2c143088e6034d52b6cfa04] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=35.5 K 2023-03-30 03:56:20,315 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5f95bdd6dc7041e0b65df8137cd18188, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/13922bdaca7b4f26a45054a67b874b6c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d5dfed8f977d4ab5adfe053b3bd79569] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=35.5 K 2023-03-30 03:56:20,316 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5f95bdd6dc7041e0b65df8137cd18188, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680148578439 2023-03-30 03:56:20,316 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3fdf145905e449c19d5298e91e8679e0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680148578439 2023-03-30 03:56:20,319 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 13922bdaca7b4f26a45054a67b874b6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680148579080 2023-03-30 03:56:20,319 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,319 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,320 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ba7fc0292eac42bcad19e0c382ddc397, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680148579080 2023-03-30 03:56:20,320 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 04058b3fe2c143088e6034d52b6cfa04, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680148579425 2023-03-30 03:56:20,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:20,321 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting d5dfed8f977d4ab5adfe053b3bd79569, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680148579425 2023-03-30 03:56:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27fb2232[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:20,326 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:20,326 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,327 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:20,327 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:20,327 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:20,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:20,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:20,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:20,329 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,341 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:20,341 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:20,341 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:20,341 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:20,341 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:20,341 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:20,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,345 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,347 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,352 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,352 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,352 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:20,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,352 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,352 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,353 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:20,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,353 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:20,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,402 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/2aa0e6f058ab4a9db0a955cf8eca845c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2aa0e6f058ab4a9db0a955cf8eca845c 2023-03-30 03:56:20,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:20,410 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-30 03:56:20,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:20,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:20,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:20,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:20,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:20,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:20,412 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into 2aa0e6f058ab4a9db0a955cf8eca845c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:20,412 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:20,412 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148580313; duration=0sec 2023-03-30 03:56:20,413 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:20,413 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:20,414 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:20,415 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:20,415 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,415 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c7ba2d54bafe48f88d133401d2d580f1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ae8670823e6342978de61f25f808441a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/38139219f31d4e22b21f1467c34af3ae] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=35.5 K 2023-03-30 03:56:20,416 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c7ba2d54bafe48f88d133401d2d580f1, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680148578439 2023-03-30 03:56:20,416 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ae8670823e6342978de61f25f808441a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680148579080 2023-03-30 03:56:20,417 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 38139219f31d4e22b21f1467c34af3ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680148579425 2023-03-30 03:56:20,453 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:20,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:20,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148640448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148640451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,465 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:20,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148640459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148640457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:20,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:20,477 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:20,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/1c531974ffbd4829ac756106236f6894 2023-03-30 03:56:20,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:20,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:20,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,481 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,481 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:20,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:20,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:20,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:20,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:20,535 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:20,535 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:20,536 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,536 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:20,536 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:20,536 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:20,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:20,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:20,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:20,538 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,538 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,538 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,539 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff80cf4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:20,540 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:20,540 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,543 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:20,543 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:20,543 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:20,543 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,543 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:20,544 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,545 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:20,545 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,545 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/e1d10e14f9754137abee87ce1f325e9d 2023-03-30 03:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:20,551 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:20,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:20,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:20,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:20,551 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:20,561 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/b590e1e07c4c4049a91389f2f95cdfd1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b590e1e07c4c4049a91389f2f95cdfd1 2023-03-30 03:56:20,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148640567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148640567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148640568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,572 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,576 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,576 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,576 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:20,576 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into b590e1e07c4c4049a91389f2f95cdfd1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:20,576 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,576 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:20,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:20,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:20,576 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,577 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:20,577 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:20,576 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148580313; duration=0sec 2023-03-30 03:56:20,577 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,577 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,577 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,577 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,577 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,577 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:20,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148640578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,677 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:20,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:20,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:20,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:20,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:20,690 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:20,690 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,690 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:20,690 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:20,691 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,691 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,692 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,692 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,693 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,693 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:20,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:20,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:20,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:20,708 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:20,709 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,709 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:20,709 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:20,709 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:20,711 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,711 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,712 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,712 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:20,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1586ff62[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:20,713 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:20,713 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,715 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,715 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:20,727 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,727 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:20,727 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:20,727 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:20,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,729 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:20,729 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,729 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:20,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:20,732 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:20,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:20,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:20,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:20,732 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:20,733 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,733 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,733 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,734 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,736 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,741 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,741 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,741 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,741 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,741 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,741 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,741 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,741 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:20,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,741 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:20,742 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:20,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:20,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148640772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148640774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148640774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,791 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/26b0e9c65e25446897963ed56e4a6565 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/26b0e9c65e25446897963ed56e4a6565 2023-03-30 03:56:20,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:20,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148640791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,802 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into 26b0e9c65e25446897963ed56e4a6565(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:20,802 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:20,803 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148580313; duration=0sec 2023-03-30 03:56:20,803 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:20,842 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:20,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:20,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:20,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:20,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:20,854 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:20,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:20,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:20,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,855 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,856 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,856 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:20,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:20,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:20,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:20,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:20,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:20,862 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,862 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:20,862 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:20,862 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,863 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:20,863 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:20,868 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@424b1ba7[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:20,899 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:20,900 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,901 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,901 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:20,901 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:20,901 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:20,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:20,904 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,904 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,904 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,906 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:20,909 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:20,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:20,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:20,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:20,909 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:20,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,911 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:20,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:20,912 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:20,918 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:20,918 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:20,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:20,918 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,918 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,919 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:20,919 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:20,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:20,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:20,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:20,919 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,919 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:20,919 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:20,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:20,919 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:20,919 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:20,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:21,020 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:21,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/173d4ee5bae64920b2ff326f7351f0f2 2023-03-30 03:56:21,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:21,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:21,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:21,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:21,046 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:21,046 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,046 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:21,046 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:21,047 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,047 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/1c531974ffbd4829ac756106236f6894 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1c531974ffbd4829ac756106236f6894 2023-03-30 03:56:21,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,051 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,051 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:21,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:21,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:21,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:21,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:21,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:21,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:21,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:21,061 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,061 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:21,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:21,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:21,063 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:21,063 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,063 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:21,063 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:21,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1c531974ffbd4829ac756106236f6894, entries=250, sequenceid=181, filesize=16.4 K 2023-03-30 03:56:21,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,067 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,068 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/e1d10e14f9754137abee87ce1f325e9d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e1d10e14f9754137abee87ce1f325e9d 2023-03-30 03:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c5f5b6e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:21,073 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:21,073 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,076 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,076 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:21,076 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:21,076 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:21,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:21,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148641077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:21,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:21,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:21,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:21,080 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:21,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:21,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:21,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:21,080 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:21,081 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,081 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,082 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,083 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,084 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e1d10e14f9754137abee87ce1f325e9d, entries=150, sequenceid=181, filesize=11.8 K 2023-03-30 03:56:21,086 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,086 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:21,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:21,086 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,087 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:21,087 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:21,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,088 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148641087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/173d4ee5bae64920b2ff326f7351f0f2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/173d4ee5bae64920b2ff326f7351f0f2 2023-03-30 03:56:21,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148641087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/173d4ee5bae64920b2ff326f7351f0f2, entries=150, sequenceid=181, filesize=11.8 K 2023-03-30 03:56:21,097 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 3ecc6181b1966d72d93cc5e543ef979f in 688ms, sequenceid=181, compaction requested=false 2023-03-30 03:56:21,097 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:21,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:21,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 03:56:21,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:21,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:21,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:21,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:21,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:21,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:21,139 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/51cd6a92c9354091b9967fe46bd122ee 2023-03-30 03:56:21,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148641165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6dc52327532e44688eb3a42029a2be33 2023-03-30 03:56:21,187 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:21,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:21,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:21,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:21,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:21,192 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:21,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:21,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:21,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,193 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:21,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:21,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:21,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:21,196 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,196 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:21,196 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:21,196 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:21,196 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:21,197 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,197 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@631f6fbd[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:21,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:21,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,201 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,201 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:21,201 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:21,201 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:21,203 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,203 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:21,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:21,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:21,204 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:21,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:21,206 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:21,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:21,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:21,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:21,206 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:21,207 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,210 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,211 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,218 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:21,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,226 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:21,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:21,226 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:21,226 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:21,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148641268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,326 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:21,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:21,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:21,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:21,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:21,330 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:21,330 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,330 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:21,330 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:21,331 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,331 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,331 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,332 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,332 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:21,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:21,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:21,333 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,333 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:21,333 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:21,333 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:21,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:21,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:21,337 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:21,337 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4140e153[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:21,338 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:21,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:21,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:21,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:21,339 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:21,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,339 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,339 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:21,343 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:21,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:21,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:21,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:21,343 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:21,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,345 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,346 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:21,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:21,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,348 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:21,348 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,448 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:21,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:21,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:21,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:21,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:21,453 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:21,453 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,453 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:21,453 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:21,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,454 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:21,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:21,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:21,456 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:21,456 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,456 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:21,459 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:21,459 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,459 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:21,459 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:21,459 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:21,459 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,459 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:21,460 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:21,460 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,460 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:21,460 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:21,461 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,461 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,461 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,461 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:21,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4c4cff[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:21,464 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:21,464 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,465 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:21,465 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:21,465 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:21,465 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,465 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:21,466 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:21,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:21,470 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:21,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:21,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:21,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:21,470 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:21,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148641476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,479 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,482 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,483 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:21,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:21,483 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,483 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,483 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,483 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,483 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:21,483 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:21,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:21,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148641580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,583 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:21,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:21,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:21,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:21,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:21,590 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:21,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:21,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:21,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148641590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,592 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148641590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,592 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:21,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:21,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:21,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:21,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:21,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:21,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:21,595 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,595 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:21,595 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:21,595 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:21,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:21,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:21,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:21,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/79ba1dbd6a79414390c067bfbda29349 2023-03-30 03:56:21,597 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/51cd6a92c9354091b9967fe46bd122ee as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/51cd6a92c9354091b9967fe46bd122ee 2023-03-30 03:56:21,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:21,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0ee8d3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:21,609 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:21,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,611 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:21,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:21,611 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:21,611 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:21,612 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,612 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,612 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:21,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:21,616 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:21,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:21,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:21,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:21,616 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:21,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/51cd6a92c9354091b9967fe46bd122ee, entries=150, sequenceid=203, filesize=11.8 K 2023-03-30 03:56:21,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6dc52327532e44688eb3a42029a2be33 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6dc52327532e44688eb3a42029a2be33 2023-03-30 03:56:21,626 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,626 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:21,626 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:21,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,627 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6dc52327532e44688eb3a42029a2be33, entries=150, sequenceid=203, filesize=11.8 K 2023-03-30 03:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:21,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/79ba1dbd6a79414390c067bfbda29349 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/79ba1dbd6a79414390c067bfbda29349 2023-03-30 03:56:21,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:21,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:21,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:21,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:21,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:21,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:21,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,638 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:21,638 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:21,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/79ba1dbd6a79414390c067bfbda29349, entries=150, sequenceid=203, filesize=11.8 K 2023-03-30 03:56:21,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3ecc6181b1966d72d93cc5e543ef979f in 533ms, sequenceid=203, compaction requested=true 2023-03-30 03:56:21,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:21,644 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:21,644 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:21,644 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:21,644 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:21,645 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:21,654 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41298 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:21,654 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:21,655 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,655 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/26b0e9c65e25446897963ed56e4a6565, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1c531974ffbd4829ac756106236f6894, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/51cd6a92c9354091b9967fe46bd122ee] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=40.3 K 2023-03-30 03:56:21,655 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:21,655 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:21,655 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,655 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2aa0e6f058ab4a9db0a955cf8eca845c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e1d10e14f9754137abee87ce1f325e9d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6dc52327532e44688eb3a42029a2be33] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=35.7 K 2023-03-30 03:56:21,656 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2aa0e6f058ab4a9db0a955cf8eca845c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680148579425 2023-03-30 03:56:21,656 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 26b0e9c65e25446897963ed56e4a6565, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680148579425 2023-03-30 03:56:21,656 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e1d10e14f9754137abee87ce1f325e9d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148580088 2023-03-30 03:56:21,657 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1c531974ffbd4829ac756106236f6894, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148580085 2023-03-30 03:56:21,657 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6dc52327532e44688eb3a42029a2be33, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680148580443 2023-03-30 03:56:21,659 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 51cd6a92c9354091b9967fe46bd122ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680148580443 2023-03-30 03:56:21,668 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:21,671 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:21,704 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/55c7b64677474e39ad2c5f3985028a0c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/55c7b64677474e39ad2c5f3985028a0c 2023-03-30 03:56:21,708 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/7861088115f74d6cb96faae801c4f35c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7861088115f74d6cb96faae801c4f35c 2023-03-30 03:56:21,715 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into 55c7b64677474e39ad2c5f3985028a0c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:21,715 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:21,715 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into 7861088115f74d6cb96faae801c4f35c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:21,715 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148581644; duration=0sec 2023-03-30 03:56:21,715 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:21,715 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148581644; duration=0sec 2023-03-30 03:56:21,715 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:21,715 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:21,715 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:21,717 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:21,717 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:21,717 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,717 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b590e1e07c4c4049a91389f2f95cdfd1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/173d4ee5bae64920b2ff326f7351f0f2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/79ba1dbd6a79414390c067bfbda29349] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=35.7 K 2023-03-30 03:56:21,717 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b590e1e07c4c4049a91389f2f95cdfd1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680148579425 2023-03-30 03:56:21,718 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 173d4ee5bae64920b2ff326f7351f0f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148580088 2023-03-30 03:56:21,718 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 79ba1dbd6a79414390c067bfbda29349, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680148580443 2023-03-30 03:56:21,736 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#47 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:21,738 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:21,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:21,748 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:21,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:21,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:21,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,750 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:21,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:21,750 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,750 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,750 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:21,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:21,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:21,762 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:21,763 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:21,763 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:56:21,763 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:21,763 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:21,763 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:21,763 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:21,763 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:21,763 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:21,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:21,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:21,792 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/74abe3a1a0a64c009275ef6a1381b6d9 2023-03-30 03:56:21,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148641822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:21,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:21,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37654 deadline: 1680148641924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148642128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,169 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/3bcca592da2c41e1ab8e8fbd5a21e715 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3bcca592da2c41e1ab8e8fbd5a21e715 2023-03-30 03:56:22,181 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into 3bcca592da2c41e1ab8e8fbd5a21e715(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:22,181 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:22,181 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148581645; duration=0sec 2023-03-30 03:56:22,181 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:22,209 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6b1ee89a0a7b442b86b9146462630cd6 2023-03-30 03:56:22,236 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/ee533cb1149b4520816f24e3373833e5 2023-03-30 03:56:22,243 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/74abe3a1a0a64c009275ef6a1381b6d9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/74abe3a1a0a64c009275ef6a1381b6d9 2023-03-30 03:56:22,250 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/74abe3a1a0a64c009275ef6a1381b6d9, entries=150, sequenceid=218, filesize=11.8 K 2023-03-30 03:56:22,251 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6b1ee89a0a7b442b86b9146462630cd6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6b1ee89a0a7b442b86b9146462630cd6 2023-03-30 03:56:22,265 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6b1ee89a0a7b442b86b9146462630cd6, entries=150, sequenceid=218, filesize=11.8 K 2023-03-30 03:56:22,266 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/ee533cb1149b4520816f24e3373833e5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ee533cb1149b4520816f24e3373833e5 2023-03-30 03:56:22,274 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ee533cb1149b4520816f24e3373833e5, entries=150, sequenceid=218, filesize=11.8 K 2023-03-30 03:56:22,275 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3ecc6181b1966d72d93cc5e543ef979f in 512ms, sequenceid=218, compaction requested=false 2023-03-30 03:56:22,275 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:22,275 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,275 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:22,275 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:22,276 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,276 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:22,276 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:22,277 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,277 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,277 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,277 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:22,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5741e48[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:22,279 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:22,279 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,281 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:22,281 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,281 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:22,282 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:22,282 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,282 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:22,285 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:22,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:22,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:22,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:22,286 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:22,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:22,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:22,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,294 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:22,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:22,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:22,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:22,294 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:22,294 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:22,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:22,395 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:22,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:22,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:22,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:22,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:22,404 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:22,404 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:22,404 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:22,404 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:22,406 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,406 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:22,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:22,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:22,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:22,414 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:22,414 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:22,414 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:22,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:22,418 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:22,418 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:22,418 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 03:56:22,418 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:22,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:22,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:22,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,435 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/c9e5bb577a6d4fb49545a9238369d2e6 2023-03-30 03:56:22,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:22,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:22,477 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/97ed502d8aef4d229937f9f19c16e2b6 2023-03-30 03:56:22,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148642487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,560 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/b488791e97c942f4ad76c7c41058abe5 2023-03-30 03:56:22,570 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/c9e5bb577a6d4fb49545a9238369d2e6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/c9e5bb577a6d4fb49545a9238369d2e6 2023-03-30 03:56:22,579 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/c9e5bb577a6d4fb49545a9238369d2e6, entries=150, sequenceid=238, filesize=11.8 K 2023-03-30 03:56:22,581 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/97ed502d8aef4d229937f9f19c16e2b6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/97ed502d8aef4d229937f9f19c16e2b6 2023-03-30 03:56:22,589 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/97ed502d8aef4d229937f9f19c16e2b6, entries=150, sequenceid=238, filesize=11.8 K 2023-03-30 03:56:22,591 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/b488791e97c942f4ad76c7c41058abe5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b488791e97c942f4ad76c7c41058abe5 2023-03-30 03:56:22,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148642588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,612 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b488791e97c942f4ad76c7c41058abe5, entries=150, sequenceid=238, filesize=11.8 K 2023-03-30 03:56:22,613 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3ecc6181b1966d72d93cc5e543ef979f in 195ms, sequenceid=238, compaction requested=true 2023-03-30 03:56:22,613 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:22,613 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:22,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:22,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:22,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:22,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:22,616 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 03:56:22,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:22,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:22,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:22,620 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,620 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,620 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,620 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,620 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:22,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f83dcc[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:22,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:22,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,626 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:22,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:22,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:22,626 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,626 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:22,628 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,628 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,628 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:22,628 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:22,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:22,631 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:22,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:22,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:22,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:22,632 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:22,636 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:22,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:22,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:22,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:22,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:22,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:22,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:22,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:22,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:22,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:22,643 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:22,643 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:22,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/aa476678f56143dfaaf6c9972b058405 2023-03-30 03:56:22,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148642653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148642656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148642658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/9dfd58958a954b4bb183236033b7faed 2023-03-30 03:56:22,744 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:22,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:22,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:22,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148642760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148642761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:22,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:22,768 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:22,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:22,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:22,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:22,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,769 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:22,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:22,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148642769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:22,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:22,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:22,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:22,804 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:22,804 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:22,804 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:22,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/ce116fe7313f40f59890628ac2886305 2023-03-30 03:56:22,813 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:22,813 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:22,813 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:22,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/aa476678f56143dfaaf6c9972b058405 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/aa476678f56143dfaaf6c9972b058405 2023-03-30 03:56:22,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:22,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:22,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:22,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:22,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,816 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:22,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aeaaaf8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:22,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:22,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:22,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:22,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:22,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/aa476678f56143dfaaf6c9972b058405, entries=250, sequenceid=254, filesize=16.4 K 2023-03-30 03:56:22,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:22,824 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/9dfd58958a954b4bb183236033b7faed as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/9dfd58958a954b4bb183236033b7faed 2023-03-30 03:56:22,824 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:22,824 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:22,824 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:22,824 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:22,827 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:22,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:22,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:22,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:22,828 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:22,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:22,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:22,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,830 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,831 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/9dfd58958a954b4bb183236033b7faed, entries=150, sequenceid=254, filesize=11.8 K 2023-03-30 03:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,831 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/ce116fe7313f40f59890628ac2886305 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ce116fe7313f40f59890628ac2886305 2023-03-30 03:56:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,834 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:22,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:22,834 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,835 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,835 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,835 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:22,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:22,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:22,835 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,835 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,835 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,835 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:22,835 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:22,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:22,835 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:22,835 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:22,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ce116fe7313f40f59890628ac2886305, entries=150, sequenceid=254, filesize=11.8 K 2023-03-30 03:56:22,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 3ecc6181b1966d72d93cc5e543ef979f in 250ms, sequenceid=254, compaction requested=true 2023-03-30 03:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:22,866 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:22,866 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:22,869 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53436 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:22,869 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:22,869 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48656 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:22,869 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,869 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:22,869 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7861088115f74d6cb96faae801c4f35c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/74abe3a1a0a64c009275ef6a1381b6d9, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/c9e5bb577a6d4fb49545a9238369d2e6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/aa476678f56143dfaaf6c9972b058405] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=52.2 K 2023-03-30 03:56:22,869 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,869 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/55c7b64677474e39ad2c5f3985028a0c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6b1ee89a0a7b442b86b9146462630cd6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/97ed502d8aef4d229937f9f19c16e2b6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/9dfd58958a954b4bb183236033b7faed] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=47.5 K 2023-03-30 03:56:22,870 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 7861088115f74d6cb96faae801c4f35c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680148580443 2023-03-30 03:56:22,870 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 55c7b64677474e39ad2c5f3985028a0c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680148580443 2023-03-30 03:56:22,870 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 74abe3a1a0a64c009275ef6a1381b6d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148581147 2023-03-30 03:56:22,870 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6b1ee89a0a7b442b86b9146462630cd6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148581147 2023-03-30 03:56:22,870 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting c9e5bb577a6d4fb49545a9238369d2e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680148581781 2023-03-30 03:56:22,871 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 97ed502d8aef4d229937f9f19c16e2b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680148581781 2023-03-30 03:56:22,871 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting aa476678f56143dfaaf6c9972b058405, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680148582475 2023-03-30 03:56:22,871 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 9dfd58958a954b4bb183236033b7faed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680148582478 2023-03-30 03:56:22,888 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:22,892 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#58 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:22,935 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:22,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:22,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:22,950 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/d3daea26028649be89e9b3134df757c1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d3daea26028649be89e9b3134df757c1 2023-03-30 03:56:22,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:22,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:22,951 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:22,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:22,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:22,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:22,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,952 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:22,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:22,952 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:22,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:22,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:22,961 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into d3daea26028649be89e9b3134df757c1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:22,961 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:22,961 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=12, startTime=1680148582866; duration=0sec 2023-03-30 03:56:22,961 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:22,961 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:22,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:22,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:22,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:22,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:22,967 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48656 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:22,968 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:22,968 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,969 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3bcca592da2c41e1ab8e8fbd5a21e715, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ee533cb1149b4520816f24e3373833e5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b488791e97c942f4ad76c7c41058abe5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ce116fe7313f40f59890628ac2886305] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=47.5 K 2023-03-30 03:56:22,969 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3bcca592da2c41e1ab8e8fbd5a21e715, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680148580443 2023-03-30 03:56:22,970 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ee533cb1149b4520816f24e3373833e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148581147 2023-03-30 03:56:22,970 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b488791e97c942f4ad76c7c41058abe5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680148581781 2023-03-30 03:56:22,971 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ce116fe7313f40f59890628ac2886305, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680148582478 2023-03-30 03:56:22,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:22,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 03:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:22,979 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,979 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:22,979 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:22,979 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:22,981 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,981 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,981 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,981 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:22,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10489d06[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:22,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:22,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:22,984 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:22,984 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:22,984 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:22,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:22,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:22,985 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:22,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:22,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:22,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:22,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:22,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:22,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:22,991 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:22,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:22,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:22,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:22,991 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:23,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,003 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,005 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:23,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:23,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,010 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:23,010 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:23,009 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:23,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,022 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:23,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148643025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148643027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148643028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/2b7e48556973481d81f8b6dee44379bc 2023-03-30 03:56:23,069 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/cc8daf17389e4ba19d47ac7264ebc3cc as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/cc8daf17389e4ba19d47ac7264ebc3cc 2023-03-30 03:56:23,080 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into cc8daf17389e4ba19d47ac7264ebc3cc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:23,081 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:23,081 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=12, startTime=1680148582866; duration=0sec 2023-03-30 03:56:23,081 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:23,110 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:23,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:23,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:23,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:23,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:23,118 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:23,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:23,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:23,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,119 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,119 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:23,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:23,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:23,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:23,130 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:23,130 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:23,130 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,130 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:23,130 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:23,131 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:23,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:23,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:23,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:23,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148643131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:23,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35eb5719[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:23,134 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:23,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:23,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:23,144 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:23,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148643143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:23,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:23,145 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148643144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,148 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,148 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:23,150 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:23,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:23,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:23,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:23,150 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:23,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,152 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,154 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,176 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,177 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:23,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:23,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,177 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:23,177 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:23,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,277 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:23,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:23,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:23,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:23,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:23,282 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:23,282 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,282 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:23,282 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:23,282 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,282 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,283 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,284 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,284 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:23,284 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:23,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:23,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:23,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:23,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:23,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:23,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:23,319 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,319 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:23,320 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:23,320 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:23,321 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:23,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377f0a35[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:23,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:23,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,324 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:23,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:23,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:23,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:23,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:23,329 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:23,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:23,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:23,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:23,329 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:23,330 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,330 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,330 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,331 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,333 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148643334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:23,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:23,344 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:23,344 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148643346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148643349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,367 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/ec159cac0ec340c58e9b89323b6fe4eb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec159cac0ec340c58e9b89323b6fe4eb 2023-03-30 03:56:23,377 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into ec159cac0ec340c58e9b89323b6fe4eb(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:23,377 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:23,377 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=12, startTime=1680148582866; duration=0sec 2023-03-30 03:56:23,377 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:23,444 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:23,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:23,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:23,455 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:23,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:23,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:23,456 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,456 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,456 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:23,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:23,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:23,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:23,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:23,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:23,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:23,478 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,478 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:23,478 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:23,478 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:23,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:23,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:23,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:23,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,480 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,480 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,480 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18eddb48[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:23,482 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:23,482 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:23,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:23,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:23,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:23,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:23,484 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:23,488 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:23,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:23,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:23,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:23,488 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:23,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,492 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/73397c9a2a714f0a93e618d7ceeb0423 2023-03-30 03:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:23,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:23,498 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:23,498 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:23,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/1a8b4897ca4a4c398f4558e15197bc9f 2023-03-30 03:56:23,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/2b7e48556973481d81f8b6dee44379bc as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/2b7e48556973481d81f8b6dee44379bc 2023-03-30 03:56:23,598 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:23,599 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/2b7e48556973481d81f8b6dee44379bc, entries=150, sequenceid=274, filesize=11.9 K 2023-03-30 03:56:23,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:23,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/73397c9a2a714f0a93e618d7ceeb0423 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/73397c9a2a714f0a93e618d7ceeb0423 2023-03-30 03:56:23,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:23,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/73397c9a2a714f0a93e618d7ceeb0423, entries=150, sequenceid=274, filesize=11.9 K 2023-03-30 03:56:23,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:23,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:23,610 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:23,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:23,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:23,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/1a8b4897ca4a4c398f4558e15197bc9f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/1a8b4897ca4a4c398f4558e15197bc9f 2023-03-30 03:56:23,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:23,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:23,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:23,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:23,614 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,614 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:23,614 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:23,614 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:23,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:23,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,616 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:23,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c99d5fe[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:23,618 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:23,618 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,619 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,620 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:23,620 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:23,620 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:23,620 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,620 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:23,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:23,621 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/1a8b4897ca4a4c398f4558e15197bc9f, entries=150, sequenceid=274, filesize=11.9 K 2023-03-30 03:56:23,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 3ecc6181b1966d72d93cc5e543ef979f in 645ms, sequenceid=274, compaction requested=false 2023-03-30 03:56:23,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:23,641 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:23,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:23,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:23,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:23,641 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:23,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-30 03:56:23,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:23,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:23,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:23,646 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:23,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:23,652 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,652 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,653 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:23,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:23,653 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:23,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:23,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/18a3a143efdf48189d30de1ae8c06445 2023-03-30 03:56:23,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37688 deadline: 1680148643681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148643687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148643687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/d0d75ae1610c49b58bedf2505d06702d 2023-03-30 03:56:23,753 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:23,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/851d6a2378c847c8a2da3adb09cd79cb 2023-03-30 03:56:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:23,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:23,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:23,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:23,764 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:23,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:23,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:23,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/18a3a143efdf48189d30de1ae8c06445 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/18a3a143efdf48189d30de1ae8c06445 2023-03-30 03:56:23,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,765 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,765 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,767 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,767 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:23,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/18a3a143efdf48189d30de1ae8c06445, entries=200, sequenceid=291, filesize=14.3 K 2023-03-30 03:56:23,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:23,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:23,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:23,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/d0d75ae1610c49b58bedf2505d06702d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0d75ae1610c49b58bedf2505d06702d 2023-03-30 03:56:23,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:23,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:23,778 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,778 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:23,778 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:23,778 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:23,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:23,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:23,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:23,784 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,784 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,785 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,785 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0d75ae1610c49b58bedf2505d06702d, entries=150, sequenceid=291, filesize=11.9 K 2023-03-30 03:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,787 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/851d6a2378c847c8a2da3adb09cd79cb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/851d6a2378c847c8a2da3adb09cd79cb 2023-03-30 03:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@232500eb[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:23,787 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:23,787 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,788 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:23,788 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,788 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:23,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:23,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:23,789 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:23,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:23,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:23,792 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:23,792 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:23,792 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:23,792 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:23,792 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:23,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148643790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37658 deadline: 1680148643796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148643796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/851d6a2378c847c8a2da3adb09cd79cb, entries=150, sequenceid=291, filesize=11.9 K 2023-03-30 03:56:23,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 3ecc6181b1966d72d93cc5e543ef979f in 153ms, sequenceid=291, compaction requested=true 2023-03-30 03:56:23,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:23,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:23,800 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:23,800 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:23,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:23,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:23,801 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,801 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:23,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,801 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,801 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39496 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:23,801 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:23,801 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:23,801 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:23,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,802 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec159cac0ec340c58e9b89323b6fe4eb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/2b7e48556973481d81f8b6dee44379bc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/18a3a143efdf48189d30de1ae8c06445] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=38.6 K 2023-03-30 03:56:23,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,801 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:23,802 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,802 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting ec159cac0ec340c58e9b89323b6fe4eb, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680148582478 2023-03-30 03:56:23,802 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d3daea26028649be89e9b3134df757c1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/73397c9a2a714f0a93e618d7ceeb0423, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0d75ae1610c49b58bedf2505d06702d] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=36.2 K 2023-03-30 03:56:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,802 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 2b7e48556973481d81f8b6dee44379bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680148582649 2023-03-30 03:56:23,803 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d3daea26028649be89e9b3134df757c1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680148582478 2023-03-30 03:56:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:23,804 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 73397c9a2a714f0a93e618d7ceeb0423, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680148582649 2023-03-30 03:56:23,804 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 18a3a143efdf48189d30de1ae8c06445, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680148583022 2023-03-30 03:56:23,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:23,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,804 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d0d75ae1610c49b58bedf2505d06702d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680148583022 2023-03-30 03:56:23,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,809 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,809 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:23,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:23,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:23,810 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:23,810 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:23,809 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,810 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,810 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,810 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,810 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:23,810 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,809 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,810 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:23,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:23,813 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:23,815 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:23,825 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/e4de537eeff447f080e45daaf0b85cc8 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e4de537eeff447f080e45daaf0b85cc8 2023-03-30 03:56:23,831 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into e4de537eeff447f080e45daaf0b85cc8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:23,831 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:23,831 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148583800; duration=0sec 2023-03-30 03:56:23,831 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:23,831 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:23,833 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:23,833 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:23,833 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,833 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/cc8daf17389e4ba19d47ac7264ebc3cc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/1a8b4897ca4a4c398f4558e15197bc9f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/851d6a2378c847c8a2da3adb09cd79cb] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=36.2 K 2023-03-30 03:56:23,833 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting cc8daf17389e4ba19d47ac7264ebc3cc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680148582478 2023-03-30 03:56:23,834 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1a8b4897ca4a4c398f4558e15197bc9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680148582649 2023-03-30 03:56:23,834 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 851d6a2378c847c8a2da3adb09cd79cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680148583022 2023-03-30 03:56:23,841 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:23,857 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/fe6c359553e34eb2ba62464bd16e456c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fe6c359553e34eb2ba62464bd16e456c 2023-03-30 03:56:23,865 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into fe6c359553e34eb2ba62464bd16e456c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:23,865 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:23,865 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148583800; duration=0sec 2023-03-30 03:56:23,865 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:23,910 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:23,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:23,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:23,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:23,913 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:23,913 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,913 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:23,913 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:23,913 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,913 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,914 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,914 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:23,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:23,914 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:23,914 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:23,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:23,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:23,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:23,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:23,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:23,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:23,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:23,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:23,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:23,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:23,916 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-30 03:56:23,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:23,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:23,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:23,916 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:23,917 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:23,917 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:23,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:23,998 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:24,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148644011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148644014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148644015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148644112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,115 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8188 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:24,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148644116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148644118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148644118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,228 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/6e69cafda9f24fa4992cc3937bbec8fb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/6e69cafda9f24fa4992cc3937bbec8fb 2023-03-30 03:56:24,236 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into 6e69cafda9f24fa4992cc3937bbec8fb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:24,236 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:24,236 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148583800; duration=0sec 2023-03-30 03:56:24,236 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:24,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148644318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148644320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148644323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,327 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/03ef8ee055e64686bf7a6e97524c85ba 2023-03-30 03:56:24,349 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/a2eda347cdd8416ca8748481bf1d213d 2023-03-30 03:56:24,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148644603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,606 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:24,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148644621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148644622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:24,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148644624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,776 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/44942974ed5f4346879a164e730bfc58 2023-03-30 03:56:24,782 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/03ef8ee055e64686bf7a6e97524c85ba as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/03ef8ee055e64686bf7a6e97524c85ba 2023-03-30 03:56:24,788 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/03ef8ee055e64686bf7a6e97524c85ba, entries=150, sequenceid=310, filesize=11.9 K 2023-03-30 03:56:24,789 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/a2eda347cdd8416ca8748481bf1d213d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a2eda347cdd8416ca8748481bf1d213d 2023-03-30 03:56:24,794 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a2eda347cdd8416ca8748481bf1d213d, entries=150, sequenceid=310, filesize=11.9 K 2023-03-30 03:56:24,796 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/44942974ed5f4346879a164e730bfc58 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/44942974ed5f4346879a164e730bfc58 2023-03-30 03:56:24,803 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/44942974ed5f4346879a164e730bfc58, entries=150, sequenceid=310, filesize=11.9 K 2023-03-30 03:56:24,804 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 3ecc6181b1966d72d93cc5e543ef979f in 888ms, sequenceid=310, compaction requested=false 2023-03-30 03:56:24,804 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:24,804 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:24,804 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:24,804 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:24,804 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,804 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:24,804 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:24,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:24,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:24,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc48818[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:24,808 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:24,808 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:24,808 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:24,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:24,809 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:24,809 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:24,809 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,809 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:24,809 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:24,810 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:24,813 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:24,813 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:24,813 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:24,813 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:24,813 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:24,814 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:24,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,816 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,816 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,816 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:24,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,819 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:24,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:24,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:24,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:24,820 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:24,820 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:24,820 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:24,820 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:24,920 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:24,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:24,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:24,923 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:24,923 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:24,923 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:24,923 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:24,924 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,924 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:24,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:24,925 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:24,925 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:24,925 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:24,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:24,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:24,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:24,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:24,926 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:24,926 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:24,926 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:24,928 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:24,928 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:24,928 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:24,928 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:24,928 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-30 03:56:24,928 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:24,928 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:24,928 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:24,928 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:24,928 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:24,929 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:24,944 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/772275cc2b7946839aa06b2b85bd3c47 2023-03-30 03:56:24,961 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/03149bf0b8ec4bb783c4772742e6e664 2023-03-30 03:56:24,987 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/85e6adba8f3d4c1aa0d0a280e926408a 2023-03-30 03:56:24,995 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/772275cc2b7946839aa06b2b85bd3c47 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/772275cc2b7946839aa06b2b85bd3c47 2023-03-30 03:56:25,009 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/772275cc2b7946839aa06b2b85bd3c47, entries=150, sequenceid=326, filesize=11.9 K 2023-03-30 03:56:25,010 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/03149bf0b8ec4bb783c4772742e6e664 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/03149bf0b8ec4bb783c4772742e6e664 2023-03-30 03:56:25,017 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/03149bf0b8ec4bb783c4772742e6e664, entries=150, sequenceid=326, filesize=11.9 K 2023-03-30 03:56:25,018 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/85e6adba8f3d4c1aa0d0a280e926408a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/85e6adba8f3d4c1aa0d0a280e926408a 2023-03-30 03:56:25,030 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/85e6adba8f3d4c1aa0d0a280e926408a, entries=150, sequenceid=326, filesize=11.9 K 2023-03-30 03:56:25,032 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 3ecc6181b1966d72d93cc5e543ef979f in 104ms, sequenceid=326, compaction requested=true 2023-03-30 03:56:25,032 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:25,032 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:25,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:25,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:25,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:25,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:25,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a853953[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:25,039 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:25,039 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,040 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:25,040 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:25,040 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:25,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:25,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:25,042 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:25,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:25,045 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:25,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:25,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:25,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:25,045 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:25,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,053 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:25,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,059 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,060 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,060 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,060 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,060 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,060 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:25,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:25,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:25,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:25,148 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:56:25,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:25,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:25,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:25,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:25,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:25,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:25,160 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:25,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:25,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:25,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:25,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:25,174 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:25,174 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,174 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:25,174 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:25,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,176 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:25,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:25,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:25,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:25,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,200 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:25,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148585170, max timestamp=1680148585190Num uniques 859; 2023-03-30 03:56:25,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:25,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148585170, max timestamp=1680148585189Num uniques 859; 2023-03-30 03:56:25,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:25,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680148585170, max timestamp=1680148585189Num uniques 859; 2023-03-30 03:56:25,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-30 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:25,214 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,214 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:25,214 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:25,214 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:25,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:25,217 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,217 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,217 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,217 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18b139d8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:25,224 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:25,224 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,226 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,226 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:25,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:25,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:25,227 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:25,228 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,228 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:25,228 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:25,232 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:25,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:25,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:25,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:25,233 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:25,234 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,234 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,234 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,235 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,236 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,238 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,238 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,239 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,239 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,239 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,239 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,239 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148645232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,239 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:25,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:25,240 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:25,240 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:25,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680148585207, max timestamp=1680148585234Num uniques 1018; 2023-03-30 03:56:25,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 03:56:25,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680148585207, max timestamp=1680148585234Num uniques 1018; 2023-03-30 03:56:25,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 03:56:25,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680148585207, max timestamp=1680148585234Num uniques 1009; 2023-03-30 03:56:25,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-03-30 03:56:25,246 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,246 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:25,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148645296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148645295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,340 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:25,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:25,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:25,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:25,343 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:25,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:25,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:25,344 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,344 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,345 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148645341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,345 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,345 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,346 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:25,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:25,346 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:25,346 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,346 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:25,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:25,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:25,350 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,350 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:25,350 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:25,351 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,351 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:25,351 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:25,351 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,351 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:25,351 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:25,352 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,352 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,352 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,353 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1398507d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:25,354 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:25,354 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,355 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,355 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:25,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:25,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:25,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:25,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:25,361 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:25,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:25,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:25,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:25,361 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:25,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,367 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,370 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,376 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,376 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,376 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,376 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,376 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,377 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,377 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,377 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:25,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:25,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,377 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,377 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:25,377 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:25,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:25,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148645400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148645400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,477 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:25,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:25,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:25,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:25,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:25,490 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:25,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:25,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:25,491 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,491 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,491 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,492 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:25,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:25,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:25,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:25,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:25,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:25,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:25,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:25,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:25,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:25,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:25,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:25,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:25,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:25,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bd48e5e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:25,509 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:25,509 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,511 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:25,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:25,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:25,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:25,516 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:25,519 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:25,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:25,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:25,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:25,519 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:25,520 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,521 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,521 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,521 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:25,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:25,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,527 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:25,527 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:25,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:25,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148645549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/0fd2c11d904e45ef9c51d3172125e78e 2023-03-30 03:56:25,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148645606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148645606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,627 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:25,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:25,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:25,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:25,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:25,642 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:25,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:25,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:25,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,644 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,644 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:25,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:25,650 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:25,650 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,650 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:25,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:25,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:25,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/477fab27aeb44408932c0c8839cbc35a 2023-03-30 03:56:25,666 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,666 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:25,666 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:25,666 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:25,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:25,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:25,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:25,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68286bd6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:25,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:25,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,677 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,677 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:25,677 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:25,677 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:25,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:25,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,679 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:25,679 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,679 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:25,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:25,683 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:25,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:25,684 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:25,684 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:25,684 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:25,685 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,685 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,685 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,686 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,689 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,692 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,692 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,692 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:25,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:25,692 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,692 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:25,692 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:25,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,792 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:25,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:25,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:25,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:25,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:25,796 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:25,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:25,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:25,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:25,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:25,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:25,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:25,802 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:25,802 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,802 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:25,809 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:25,810 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:25,810 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,810 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:25,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:25,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:25,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:25,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:25,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:25,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,820 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,820 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7946f0f2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:25,826 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:25,826 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,827 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,827 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:25,827 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:25,827 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:25,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:25,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,828 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,828 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,828 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:25,832 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:25,832 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:25,832 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:25,832 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:25,832 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:25,842 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,843 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,843 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,846 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,852 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:25,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:25,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:25,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,852 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:25,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,853 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:25,853 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148645853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148645912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:25,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148645912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,953 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:25,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:25,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:25,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:25,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:25,964 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:25,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:25,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:25,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,966 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:25,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:25,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:25,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:25,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:25,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:25,970 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:25,970 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:25,970 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:25,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:25,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:25,986 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,986 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:25,986 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:25,986 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:25,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:25,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:25,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:25,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:25,988 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,988 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280c4c1f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:25,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:25,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:25,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:25,992 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:25,992 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:25,992 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:25,992 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:25,993 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:25,994 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:25,996 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:25,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:25,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:25,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:25,996 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:25,997 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,997 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:25,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,997 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:25,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:25,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:26,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:26,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:26,002 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:26,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,102 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:26,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:26,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/b49c3d9576544ee2986fb8835af69703 2023-03-30 03:56:26,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:26,107 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:26,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:26,108 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:26,108 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,108 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,109 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,109 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,109 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,109 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:26,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:26,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/0fd2c11d904e45ef9c51d3172125e78e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0fd2c11d904e45ef9c51d3172125e78e 2023-03-30 03:56:26,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:26,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:26,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:26,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0fd2c11d904e45ef9c51d3172125e78e, entries=250, sequenceid=336, filesize=16.7 K 2023-03-30 03:56:26,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:26,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:26,127 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,127 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:26,127 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:26,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/477fab27aeb44408932c0c8839cbc35a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/477fab27aeb44408932c0c8839cbc35a 2023-03-30 03:56:26,127 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:26,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:26,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:26,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:26,134 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/477fab27aeb44408932c0c8839cbc35a, entries=150, sequenceid=336, filesize=11.9 K 2023-03-30 03:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/b49c3d9576544ee2986fb8835af69703 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b49c3d9576544ee2986fb8835af69703 2023-03-30 03:56:26,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:26,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5755813e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:26,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:26,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,137 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:26,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:26,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:26,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:26,139 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:26,140 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,140 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:26,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b49c3d9576544ee2986fb8835af69703, entries=150, sequenceid=336, filesize=11.9 K 2023-03-30 03:56:26,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3ecc6181b1966d72d93cc5e543ef979f in 996ms, sequenceid=336, compaction requested=true 2023-03-30 03:56:26,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:26,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:26,144 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:26,144 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:26,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:26,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:26,146 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:26,146 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:26,147 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,147 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e4de537eeff447f080e45daaf0b85cc8, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a2eda347cdd8416ca8748481bf1d213d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/03149bf0b8ec4bb783c4772742e6e664, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/477fab27aeb44408932c0c8839cbc35a] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=48.3 K 2023-03-30 03:56:26,147 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54374 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:26,147 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:26,147 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,147 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/6e69cafda9f24fa4992cc3937bbec8fb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/03ef8ee055e64686bf7a6e97524c85ba, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/772275cc2b7946839aa06b2b85bd3c47, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0fd2c11d904e45ef9c51d3172125e78e] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=53.1 K 2023-03-30 03:56:26,148 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e4de537eeff447f080e45daaf0b85cc8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680148583022 2023-03-30 03:56:26,148 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 6e69cafda9f24fa4992cc3937bbec8fb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680148583022 2023-03-30 03:56:26,148 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a2eda347cdd8416ca8748481bf1d213d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680148583679 2023-03-30 03:56:26,149 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 03149bf0b8ec4bb783c4772742e6e664, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680148584010 2023-03-30 03:56:26,149 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 03ef8ee055e64686bf7a6e97524c85ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680148583679 2023-03-30 03:56:26,149 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 477fab27aeb44408932c0c8839cbc35a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680148585143 2023-03-30 03:56:26,150 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 772275cc2b7946839aa06b2b85bd3c47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680148584010 2023-03-30 03:56:26,150 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,150 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 0fd2c11d904e45ef9c51d3172125e78e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680148585133 2023-03-30 03:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:26,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:26,161 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:26,161 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:26,161 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:26,161 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:26,161 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:26,163 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,163 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,163 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,164 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,180 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#78 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:26,183 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:26,185 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,185 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,185 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,185 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,186 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,186 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,186 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,186 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,186 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:26,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:26,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,187 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:26,187 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,245 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/ac5ecd6ef7e642a7b4b02773a3c77c78 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ac5ecd6ef7e642a7b4b02773a3c77c78 2023-03-30 03:56:26,249 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/a29f237d45f548cd8eb6613e26ce0410 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a29f237d45f548cd8eb6613e26ce0410 2023-03-30 03:56:26,254 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into ac5ecd6ef7e642a7b4b02773a3c77c78(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:26,256 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:26,256 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=12, startTime=1680148586144; duration=0sec 2023-03-30 03:56:26,256 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:26,256 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:26,258 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into a29f237d45f548cd8eb6613e26ce0410(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:26,258 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:26,258 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=12, startTime=1680148586144; duration=0sec 2023-03-30 03:56:26,258 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:26,258 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:26,259 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:26,259 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,259 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fe6c359553e34eb2ba62464bd16e456c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/44942974ed5f4346879a164e730bfc58, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/85e6adba8f3d4c1aa0d0a280e926408a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b49c3d9576544ee2986fb8835af69703] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=48.3 K 2023-03-30 03:56:26,259 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting fe6c359553e34eb2ba62464bd16e456c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1680148583022 2023-03-30 03:56:26,259 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 44942974ed5f4346879a164e730bfc58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680148583679 2023-03-30 03:56:26,260 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 85e6adba8f3d4c1aa0d0a280e926408a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680148584010 2023-03-30 03:56:26,260 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b49c3d9576544ee2986fb8835af69703, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680148585143 2023-03-30 03:56:26,281 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#80 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:26,287 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:26,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:26,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:26,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:26,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:26,298 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:26,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:26,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:26,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,300 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:26,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:26,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:26,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:26,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:26,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:26,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:26,302 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,302 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:26,302 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-03-30 03:56:26,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:26,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:26,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:26,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:26,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:26,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:26,311 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/f958a8cc0c7844088c775a9d91af77cd as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/f958a8cc0c7844088c775a9d91af77cd 2023-03-30 03:56:26,335 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into f958a8cc0c7844088c775a9d91af77cd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:26,335 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:26,335 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=12, startTime=1680148586144; duration=0sec 2023-03-30 03:56:26,335 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:26,348 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/4f409dfba5024dd8ba9cb33f7ae77852 2023-03-30 03:56:26,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:26,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:26,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:26,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148646390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,393 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/531e5331e00841af90ddebbbb0efeb30 2023-03-30 03:56:26,410 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/3c78920eccf04dc4ba0a90d215d9d48e 2023-03-30 03:56:26,417 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/4f409dfba5024dd8ba9cb33f7ae77852 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/4f409dfba5024dd8ba9cb33f7ae77852 2023-03-30 03:56:26,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:26,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148646418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:26,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148646421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,422 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/4f409dfba5024dd8ba9cb33f7ae77852, entries=150, sequenceid=391, filesize=11.9 K 2023-03-30 03:56:26,423 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/531e5331e00841af90ddebbbb0efeb30 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/531e5331e00841af90ddebbbb0efeb30 2023-03-30 03:56:26,429 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/531e5331e00841af90ddebbbb0efeb30, entries=150, sequenceid=391, filesize=11.9 K 2023-03-30 03:56:26,430 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/3c78920eccf04dc4ba0a90d215d9d48e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3c78920eccf04dc4ba0a90d215d9d48e 2023-03-30 03:56:26,435 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3c78920eccf04dc4ba0a90d215d9d48e, entries=150, sequenceid=391, filesize=11.9 K 2023-03-30 03:56:26,436 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for 3ecc6181b1966d72d93cc5e543ef979f in 134ms, sequenceid=391, compaction requested=false 2023-03-30 03:56:26,436 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:26,436 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:26,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:26,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:26,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:26,437 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,437 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8826d9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:26,439 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:26,439 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:26,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:26,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:26,440 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,441 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:26,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:26,443 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:26,443 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:26,444 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:26,444 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:26,444 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:26,445 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,445 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,445 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,445 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,448 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,450 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:26,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:26,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,450 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:26,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 03:56:26,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:26,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:26,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:26,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:26,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:26,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:26,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/5ea960a4861442fb9b14228de60da4a9 2023-03-30 03:56:26,550 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:26,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:26,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:26,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:26,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:26,553 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:26,553 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,553 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:26,553 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:26,554 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,554 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,554 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:26,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:26,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:26,556 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:26,556 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,556 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:26,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:26,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:26,558 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,558 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:26,558 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:26,558 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:26,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:26,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:26,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:26,559 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,559 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,559 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,560 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:26,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@328798da[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:26,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:26,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,563 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,563 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:26,563 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:26,563 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:26,563 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,563 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:26,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:26,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,564 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:26,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:26,566 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:26,566 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:26,566 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:26,566 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:26,566 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:26,567 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,567 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,567 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,568 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,569 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,570 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,570 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,570 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:26,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:26,570 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,570 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,571 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:26,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:26,571 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:26,571 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,571 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,571 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,571 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148586550, max timestamp=1680148586573Num uniques 859; 2023-03-30 03:56:26,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:26,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148586550, max timestamp=1680148586573Num uniques 859; 2023-03-30 03:56:26,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:26,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148586550, max timestamp=1680148586573Num uniques 859; 2023-03-30 03:56:26,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:26,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:26,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148646658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,671 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:26,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:26,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:26,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:26,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:26,677 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:26,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:26,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:26,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,678 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,679 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,679 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:26,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:26,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:26,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:26,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:26,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:26,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:26,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:26,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:26,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:26,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:26,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:26,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:26,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:26,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:26,693 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,693 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,693 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:26,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78214621[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:26,695 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:26,696 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,696 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:26,696 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:26,696 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:26,702 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,702 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:26,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:26,702 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:26,705 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:26,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:26,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:26,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:26,705 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:26,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,707 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,712 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:26,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:26,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,712 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:26,712 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:26,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148646762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,812 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:26,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:26,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:26,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:26,821 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:26,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:26,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:26,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,822 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:26,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:26,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:26,834 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:26,834 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,834 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:26,834 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:26,834 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:26,834 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,834 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:26,835 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:26,835 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:26,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:26,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:26,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:26,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:26,843 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,843 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,843 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,843 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:26,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386746a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:26,845 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:26,845 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,846 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:26,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:26,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:26,846 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,846 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:26,847 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:26,847 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,847 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,847 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:26,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:26,850 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:26,850 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:26,850 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:26,850 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:26,850 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:26,851 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,851 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,851 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:26,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,852 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:26,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:26,856 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,856 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:26,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:26,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:26,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:26,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:26,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:26,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:26,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/feba0d7988ec439aaffad3bdb6ca7945 2023-03-30 03:56:26,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:26,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:26,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:26,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:26,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:26,974 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:26,974 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:26,975 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:26,975 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:26,975 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,975 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,976 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:26,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:26,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:26,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:26,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:26,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:26,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148646974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:27,003 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:27,003 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,003 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:27,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:27,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:27,006 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,006 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:27,006 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:27,006 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:27,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:27,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:27,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:27,007 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,007 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e85242d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:27,009 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:27,009 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:27,010 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:27,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:27,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:27,014 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:27,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:27,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:27,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:27,014 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:27,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:27,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:27,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,017 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,020 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,020 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:27,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:27,020 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:27,020 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:27,021 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:27,020 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,021 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:27,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:27,021 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:27,021 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:27,021 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,021 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,021 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:27,121 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:27,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:27,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:27,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:27,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:27,128 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:27,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:27,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:27,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,130 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:27,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:27,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:27,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:27,134 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,134 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:27,134 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:27,134 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:27,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:27,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:27,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:27,136 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,136 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,136 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,136 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:27,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30ca3a6f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:27,138 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:27,138 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,139 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,139 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:27,139 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:27,139 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:27,140 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,140 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:27,140 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:27,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:27,143 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:27,143 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:27,143 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:27,143 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:27,143 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:27,145 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,146 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,147 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,148 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:27,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,148 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:27,148 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:27,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:27,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:27,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:27,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:27,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:27,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:27,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:27,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:27,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,150 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,150 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,150 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:27,150 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:27,150 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:27,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:27,250 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:27,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:27,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:27,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:27,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:27,254 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:27,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:27,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:27,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,255 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:27,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:27,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:27,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:27,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:27,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:27,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:27,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:27,257 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,257 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:27,257 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:27,257 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:27,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:27,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:27,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:27,258 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:27,260 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42287219[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:27,260 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,260 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,261 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:27,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:27,261 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:27,261 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:27,261 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:27,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:27,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:27,264 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:27,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:27,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:27,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:27,264 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:27,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:27,268 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,276 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:27,276 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:27,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:27,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:27,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:27,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:27,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:27,276 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:27,276 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,276 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:27,276 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,276 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,276 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,276 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:27,277 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:27,277 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:27,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:27,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148647284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,377 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:27,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:27,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:27,380 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:27,380 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,380 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:27,380 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:27,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,382 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,382 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:27,382 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:27,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:27,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:27,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:27,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:27,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:27,385 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:27,385 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:27,385 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:27,385 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:27,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:27,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:27,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:27,386 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,386 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,387 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,387 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723d7b12[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:27,388 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:27,388 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,389 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,389 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:27,389 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:27,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:27,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:27,390 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,390 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,390 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:27,390 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:27,393 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:27,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:27,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:27,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:27,394 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:27,395 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,395 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:27,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,395 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:27,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:27,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,396 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:27,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:27,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:27,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:27,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:27,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:27,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:27,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:27,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:27,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,401 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:27,401 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:27,413 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/d83bd16d7f58468490ef07f9c90d02ae 2023-03-30 03:56:27,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/5ea960a4861442fb9b14228de60da4a9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5ea960a4861442fb9b14228de60da4a9 2023-03-30 03:56:27,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148647431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:27,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148647432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5ea960a4861442fb9b14228de60da4a9, entries=200, sequenceid=403, filesize=14.3 K 2023-03-30 03:56:27,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/feba0d7988ec439aaffad3bdb6ca7945 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/feba0d7988ec439aaffad3bdb6ca7945 2023-03-30 03:56:27,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/feba0d7988ec439aaffad3bdb6ca7945, entries=150, sequenceid=403, filesize=11.9 K 2023-03-30 03:56:27,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/d83bd16d7f58468490ef07f9c90d02ae as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d83bd16d7f58468490ef07f9c90d02ae 2023-03-30 03:56:27,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d83bd16d7f58468490ef07f9c90d02ae, entries=150, sequenceid=403, filesize=11.9 K 2023-03-30 03:56:27,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3ecc6181b1966d72d93cc5e543ef979f in 964ms, sequenceid=403, compaction requested=true 2023-03-30 03:56:27,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:27,461 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:27,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:27,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:27,461 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:27,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:27,464 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:27,464 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:27,464 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:27,464 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:27,465 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,465 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,465 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ac5ecd6ef7e642a7b4b02773a3c77c78, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/531e5331e00841af90ddebbbb0efeb30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/feba0d7988ec439aaffad3bdb6ca7945] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=36.6 K 2023-03-30 03:56:27,465 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a29f237d45f548cd8eb6613e26ce0410, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/4f409dfba5024dd8ba9cb33f7ae77852, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5ea960a4861442fb9b14228de60da4a9] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=38.9 K 2023-03-30 03:56:27,465 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ac5ecd6ef7e642a7b4b02773a3c77c78, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680148585143 2023-03-30 03:56:27,465 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting a29f237d45f548cd8eb6613e26ce0410, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680148585143 2023-03-30 03:56:27,465 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 531e5331e00841af90ddebbbb0efeb30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680148585289 2023-03-30 03:56:27,466 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 4f409dfba5024dd8ba9cb33f7ae77852, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680148585289 2023-03-30 03:56:27,466 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting feba0d7988ec439aaffad3bdb6ca7945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680148586362 2023-03-30 03:56:27,466 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5ea960a4861442fb9b14228de60da4a9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680148586362 2023-03-30 03:56:27,476 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:27,476 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:27,501 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:27,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:27,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:27,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:27,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:27,506 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:27,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:27,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:27,507 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,507 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,507 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,507 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:27,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:27,508 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,508 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:27,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:27,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:27,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:27,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:27,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:27,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:27,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:27,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:27,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:27,522 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-30 03:56:27,523 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:27,523 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:27,523 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:27,523 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:27,524 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:27,524 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:27,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:27,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:27,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:27,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148647827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,906 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/fd8e21f31de6420d945246e631d26f92 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fd8e21f31de6420d945246e631d26f92 2023-03-30 03:56:27,907 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/705c8032cbc14a119d5342814816b3ed as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/705c8032cbc14a119d5342814816b3ed 2023-03-30 03:56:27,917 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into 705c8032cbc14a119d5342814816b3ed(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:27,917 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into fd8e21f31de6420d945246e631d26f92(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:27,917 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:27,917 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:27,917 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148587461; duration=0sec 2023-03-30 03:56:27,917 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148587461; duration=0sec 2023-03-30 03:56:27,917 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:27,917 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:27,917 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:27,918 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:27,919 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:27,919 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:27,919 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/f958a8cc0c7844088c775a9d91af77cd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3c78920eccf04dc4ba0a90d215d9d48e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d83bd16d7f58468490ef07f9c90d02ae] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=36.6 K 2023-03-30 03:56:27,920 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting f958a8cc0c7844088c775a9d91af77cd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680148585143 2023-03-30 03:56:27,920 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 3c78920eccf04dc4ba0a90d215d9d48e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680148585289 2023-03-30 03:56:27,920 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting d83bd16d7f58468490ef07f9c90d02ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680148586362 2023-03-30 03:56:27,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:27,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148647932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:27,949 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:27,954 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/0b8033a136914a9a8ab5b23f7be5960c 2023-03-30 03:56:28,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:28,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148648134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,405 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/6fb67444cd0f4925b0c3006a9519607b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/6fb67444cd0f4925b0c3006a9519607b 2023-03-30 03:56:28,411 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into 6fb67444cd0f4925b0c3006a9519607b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:28,411 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:28,411 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148587461; duration=0sec 2023-03-30 03:56:28,411 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:28,420 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/d0c2f1377638455b91f0a99834b66ed0 2023-03-30 03:56:28,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:28,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148648437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:28,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37638 deadline: 1680148648616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,619 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:28,838 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/181e7a7b223a4eee988a77bb21f152e7 2023-03-30 03:56:28,845 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/0b8033a136914a9a8ab5b23f7be5960c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0b8033a136914a9a8ab5b23f7be5960c 2023-03-30 03:56:28,852 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0b8033a136914a9a8ab5b23f7be5960c, entries=150, sequenceid=439, filesize=11.9 K 2023-03-30 03:56:28,853 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/d0c2f1377638455b91f0a99834b66ed0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0c2f1377638455b91f0a99834b66ed0 2023-03-30 03:56:28,860 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0c2f1377638455b91f0a99834b66ed0, entries=150, sequenceid=439, filesize=11.9 K 2023-03-30 03:56:28,861 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/181e7a7b223a4eee988a77bb21f152e7 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/181e7a7b223a4eee988a77bb21f152e7 2023-03-30 03:56:28,874 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/181e7a7b223a4eee988a77bb21f152e7, entries=150, sequenceid=439, filesize=11.9 K 2023-03-30 03:56:28,878 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 3ecc6181b1966d72d93cc5e543ef979f in 1357ms, sequenceid=439, compaction requested=false 2023-03-30 03:56:28,878 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:28,878 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:28,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:28,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:28,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:28,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:28,879 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:28,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:28,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:28,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:28,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:28,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650e9d34[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:28,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:28,882 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:28,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:28,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:28,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:28,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:28,883 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,883 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:28,884 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:28,884 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:28,884 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:28,884 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:28,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:28,887 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:28,887 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:28,887 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:28,887 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:28,887 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:28,889 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:28,889 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:28,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:28,889 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:28,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,896 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,901 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:28,901 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:28,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:28,901 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:28,901 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:28,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:28,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:28,901 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:28,901 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:28,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:28,901 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:28,902 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:28,902 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:28,902 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:28,902 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:28,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:28,948 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:28,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/ec76b48b513743c4b154cd813c0028e3 2023-03-30 03:56:29,002 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:29,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:29,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:29,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:29,013 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:29,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:29,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:29,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,015 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:29,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:29,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:29,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:29,028 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:29,028 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,028 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:29,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148649035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,050 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:29,050 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:29,055 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,055 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:29,056 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:29,056 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:29,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:29,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:29,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:29,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/8dca0777663c4d18b5907f77e9c657de 2023-03-30 03:56:29,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:29,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f6dab8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:29,059 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:29,059 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,061 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:29,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:29,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:29,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:29,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:29,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:29,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:29,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:29,070 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:29,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:29,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:29,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:29,070 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:29,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,085 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,085 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,086 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,094 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:29,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:29,094 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:29,094 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:29,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:29,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148649141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,194 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:29,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:29,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:29,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:29,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:29,202 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:29,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:29,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:29,203 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,203 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,204 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,204 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,204 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,205 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:29,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:29,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:29,207 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:29,207 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,207 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:29,210 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,210 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:29,210 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:29,210 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:29,218 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,218 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,218 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,218 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:29,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54fd885[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:29,219 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:29,219 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:29,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:29,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:29,226 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,226 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:29,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:29,234 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:29,236 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:29,236 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:29,237 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:29,237 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:29,237 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:29,242 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,242 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,243 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,245 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,251 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:29,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:29,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,252 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:29,252 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:29,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:29,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148649344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,352 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:29,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:29,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:29,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:29,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:29,356 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:29,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:29,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:29,357 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,357 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,358 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,358 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,358 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:29,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:29,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:29,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:29,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:29,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:29,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:29,359 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,359 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:29,360 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:29,360 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:29,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:29,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:29,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:29,361 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:29,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c33d6d5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:29,363 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:29,363 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:29,366 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:29,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:29,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:29,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:29,368 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:29,372 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:29,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:29,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:29,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:29,372 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:29,374 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,374 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,375 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:29,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:29,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:29,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,385 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:29,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,385 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:29,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:29,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148649437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,440 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:29,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:29,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148649447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,453 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:29,485 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:29,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:29,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:29,488 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:29,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:29,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:29,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:29,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:29,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:29,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:29,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:29,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:29,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:29,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:29,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:29,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:29,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:29,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,502 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,503 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,503 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d451f5a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:29,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:29,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,510 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:29,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:29,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:29,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:29,511 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:29,511 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,511 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:29,511 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:29,513 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:29,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:29,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:29,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:29,514 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:29,516 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,517 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,517 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/fa2f3799c4db4fc5b2d5c6bdb654f525 2023-03-30 03:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,531 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/ec76b48b513743c4b154cd813c0028e3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec76b48b513743c4b154cd813c0028e3 2023-03-30 03:56:29,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec76b48b513743c4b154cd813c0028e3, entries=200, sequenceid=454, filesize=14.3 K 2023-03-30 03:56:29,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/8dca0777663c4d18b5907f77e9c657de as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8dca0777663c4d18b5907f77e9c657de 2023-03-30 03:56:29,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8dca0777663c4d18b5907f77e9c657de, entries=150, sequenceid=454, filesize=11.9 K 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:29,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:29,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,552 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:29,552 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:29,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/fa2f3799c4db4fc5b2d5c6bdb654f525 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fa2f3799c4db4fc5b2d5c6bdb654f525 2023-03-30 03:56:29,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fa2f3799c4db4fc5b2d5c6bdb654f525, entries=150, sequenceid=454, filesize=11.9 K 2023-03-30 03:56:29,569 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3ecc6181b1966d72d93cc5e543ef979f in 621ms, sequenceid=454, compaction requested=true 2023-03-30 03:56:29,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:29,570 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:29,570 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:29,570 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:29,570 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:29,571 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:29,571 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:29,571 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:29,572 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,572 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/705c8032cbc14a119d5342814816b3ed, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0b8033a136914a9a8ab5b23f7be5960c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec76b48b513743c4b154cd813c0028e3] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=39.0 K 2023-03-30 03:56:29,572 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:29,572 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:29,572 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,572 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fd8e21f31de6420d945246e631d26f92, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0c2f1377638455b91f0a99834b66ed0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8dca0777663c4d18b5907f77e9c657de] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=36.7 K 2023-03-30 03:56:29,573 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 705c8032cbc14a119d5342814816b3ed, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680148586362 2023-03-30 03:56:29,573 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 0b8033a136914a9a8ab5b23f7be5960c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680148586640 2023-03-30 03:56:29,573 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting fd8e21f31de6420d945246e631d26f92, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680148586362 2023-03-30 03:56:29,573 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting ec76b48b513743c4b154cd813c0028e3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680148587817 2023-03-30 03:56:29,574 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d0c2f1377638455b91f0a99834b66ed0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680148586640 2023-03-30 03:56:29,574 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8dca0777663c4d18b5907f77e9c657de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680148587820 2023-03-30 03:56:29,607 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:29,608 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#97 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:29,647 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6626c58d46e0441aa29577ddd081c5a1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6626c58d46e0441aa29577ddd081c5a1 2023-03-30 03:56:29,652 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:29,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:29,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-30 03:56:29,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:29,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:29,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:29,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:29,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:29,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:29,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:29,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:29,667 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into 6626c58d46e0441aa29577ddd081c5a1(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:29,667 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:29,667 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148589570; duration=0sec 2023-03-30 03:56:29,667 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:29,667 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:29,674 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:29,674 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,674 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:29,675 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:29,675 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,675 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/6fb67444cd0f4925b0c3006a9519607b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/181e7a7b223a4eee988a77bb21f152e7, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fa2f3799c4db4fc5b2d5c6bdb654f525] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=36.7 K 2023-03-30 03:56:29,674 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:29,675 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:29,675 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,676 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,676 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6fb67444cd0f4925b0c3006a9519607b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680148586362 2023-03-30 03:56:29,677 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,677 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 181e7a7b223a4eee988a77bb21f152e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680148586640 2023-03-30 03:56:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:29,681 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting fa2f3799c4db4fc5b2d5c6bdb654f525, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680148587820 2023-03-30 03:56:29,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:29,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:29,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:29,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:29,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:29,690 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/9e128e9ba1c44e41a3449481d13fc965 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/9e128e9ba1c44e41a3449481d13fc965 2023-03-30 03:56:29,697 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into 9e128e9ba1c44e41a3449481d13fc965(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:29,698 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:29,698 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148589570; duration=0sec 2023-03-30 03:56:29,698 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:29,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:29,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:29,702 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,702 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:29,702 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:29,702 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:29,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:29,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148649722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,734 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:29,734 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:29,735 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:29,735 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,735 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:29,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fd42750[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:29,738 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:29,738 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,738 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:29,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:29,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:29,739 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,739 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:29,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:29,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:29,744 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:29,744 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:29,744 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:29,744 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:29,744 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:29,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,754 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,759 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,759 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:29,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:29,759 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,759 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,760 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:29,760 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:29,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:29,785 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/c169bfe01098499ebb19da01700e7f8b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c169bfe01098499ebb19da01700e7f8b 2023-03-30 03:56:29,795 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into c169bfe01098499ebb19da01700e7f8b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:29,795 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:29,795 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148589571; duration=0sec 2023-03-30 03:56:29,795 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:29,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:29,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148649827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,860 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:29,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:29,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:29,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:29,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:29,863 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:29,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:29,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:29,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:29,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:29,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:29,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:29,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:29,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:29,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:29,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:29,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:29,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:29,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:29,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:29,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:29,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:29,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:29,867 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:29,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:29,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44244bdf[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:29,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:29,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,870 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:29,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:29,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:29,870 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,870 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:29,871 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:29,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:29,873 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:29,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:29,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:29,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:29,873 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:29,874 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,874 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,874 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,875 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,876 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,878 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:29,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:29,879 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:29,879 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:29,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:29,979 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:29,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:29,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:29,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:29,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:29,982 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:29,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:29,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:29,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,983 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:29,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:29,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:29,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:29,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:29,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:29,984 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:29,985 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,985 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:29,985 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:29,985 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:29,985 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:29,986 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:29,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a3a8f0[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:29,988 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:29,988 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,988 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:29,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:29,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:29,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:29,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:29,991 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:29,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:29,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:29,993 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:29,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:29,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:29,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:29,993 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:29,994 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,994 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,994 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:29,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:29,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:29,995 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,000 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:30,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:30,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,003 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:30,003 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:30,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:30,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148650037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,103 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:30,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:30,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:30,110 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:30,110 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,110 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:30,110 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:30,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:30,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:30,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:30,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:30,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:30,123 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:30,123 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:30,123 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:30,124 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:30,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:30,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:30,124 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:30,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,126 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,126 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6339c0b1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:30,128 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:30,128 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:30,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:30,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:30,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:30,130 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:30,130 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,130 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:30,131 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:30,134 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:30,134 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:30,134 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:30,134 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:30,134 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:30,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/644a65843eca4da4a65ceeb8b8a9a58a 2023-03-30 03:56:30,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,143 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,145 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:30,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,151 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,152 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:30,152 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:30,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:30,252 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:30,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:30,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:30,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:30,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:30,255 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:30,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:30,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:30,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,257 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,257 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,257 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,257 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:30,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:30,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:30,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:30,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:30,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:30,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:30,263 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,264 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:30,264 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:30,264 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,264 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:30,264 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:30,264 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,264 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:30,264 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:30,268 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,268 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,268 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc99521[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:30,271 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:30,271 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,271 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:30,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:30,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:30,273 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,273 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:30,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:30,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:30,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:30,279 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:30,279 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:30,279 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:30,279 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:30,279 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:30,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,295 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:30,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:30,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:30,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:30,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:30,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148650340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,396 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:30,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:30,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:30,399 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:30,399 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,399 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:30,399 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:30,399 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,399 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,400 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:30,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:30,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:30,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:30,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:30,407 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,407 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:30,407 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:30,407 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:30,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:30,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b5543f4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:30,414 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:30,414 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,417 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:30,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:30,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:30,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:30,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:30,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:30,421 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:30,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:30,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:30,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:30,421 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:30,423 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,423 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,423 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,423 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,424 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,427 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:30,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,427 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,427 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:30,427 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:30,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:30,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,527 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:30,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:30,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:30,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:30,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:30,546 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:30,546 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,546 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:30,546 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:30,547 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,547 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,548 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,548 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,548 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,548 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:30,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:30,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:30,566 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:30,566 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,566 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:30,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:30,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:30,569 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/ae4d34fcd9b44cb4824c9a72c6d3e127 2023-03-30 03:56:30,570 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:30,570 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:30,570 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:30,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:30,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:30,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:30,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,598 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,598 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:30,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406a8c46[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:30,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:30,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:30,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:30,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:30,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:30,604 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:30,604 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,604 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:30,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:30,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:30,610 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:30,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:30,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:30,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:30,610 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:30,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,613 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:30,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:30,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:30,615 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:30,617 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:30,621 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:30,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:30,622 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:30,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:30,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:30,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:30,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:30,623 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:30,623 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:30,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/7f1cbee8d9c846b298dc4798ce3b3d81 2023-03-30 03:56:30,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/644a65843eca4da4a65ceeb8b8a9a58a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/644a65843eca4da4a65ceeb8b8a9a58a 2023-03-30 03:56:30,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/644a65843eca4da4a65ceeb8b8a9a58a, entries=200, sequenceid=475, filesize=14.3 K 2023-03-30 03:56:30,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/ae4d34fcd9b44cb4824c9a72c6d3e127 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ae4d34fcd9b44cb4824c9a72c6d3e127 2023-03-30 03:56:30,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ae4d34fcd9b44cb4824c9a72c6d3e127, entries=150, sequenceid=475, filesize=11.9 K 2023-03-30 03:56:30,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/7f1cbee8d9c846b298dc4798ce3b3d81 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7f1cbee8d9c846b298dc4798ce3b3d81 2023-03-30 03:56:30,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7f1cbee8d9c846b298dc4798ce3b3d81, entries=150, sequenceid=475, filesize=11.9 K 2023-03-30 03:56:30,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 3ecc6181b1966d72d93cc5e543ef979f in 1015ms, sequenceid=475, compaction requested=false 2023-03-30 03:56:30,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:30,723 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:30,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:30,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:30,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:30,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:30,732 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:30,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:30,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:30,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,733 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:30,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:30,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:30,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:30,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:30,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:30,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:30,735 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:30,735 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:30,735 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:30,737 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:30,737 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:30,737 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:30,737 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:30,737 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 03:56:30,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:30,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:30,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:30,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:30,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:30,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:30,760 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/1d231973c1e84b8b9a96d0fb09f9256e 2023-03-30 03:56:30,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:30,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:30,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148590872, max timestamp=1680148590919Num uniques 859; 2023-03-30 03:56:30,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148590872, max timestamp=1680148590919Num uniques 859; 2023-03-30 03:56:30,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:30,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148590872, max timestamp=1680148590919Num uniques 859; 2023-03-30 03:56:30,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:30,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:30,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:30,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148650981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:31,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148651086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,176 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6ab16d8d43e645869a6169d6e4cdf218 2023-03-30 03:56:31,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:31,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148651289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,587 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/b978dfc96b984ba487adaa548a9d9149 2023-03-30 03:56:31,592 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/1d231973c1e84b8b9a96d0fb09f9256e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1d231973c1e84b8b9a96d0fb09f9256e 2023-03-30 03:56:31,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:31,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148651591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,596 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1d231973c1e84b8b9a96d0fb09f9256e, entries=150, sequenceid=489, filesize=11.9 K 2023-03-30 03:56:31,597 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/6ab16d8d43e645869a6169d6e4cdf218 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ab16d8d43e645869a6169d6e4cdf218 2023-03-30 03:56:31,601 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ab16d8d43e645869a6169d6e4cdf218, entries=150, sequenceid=489, filesize=11.9 K 2023-03-30 03:56:31,602 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/b978dfc96b984ba487adaa548a9d9149 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b978dfc96b984ba487adaa548a9d9149 2023-03-30 03:56:31,606 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b978dfc96b984ba487adaa548a9d9149, entries=150, sequenceid=489, filesize=11.9 K 2023-03-30 03:56:31,607 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3ecc6181b1966d72d93cc5e543ef979f in 870ms, sequenceid=489, compaction requested=true 2023-03-30 03:56:31,607 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:31,607 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:31,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:31,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:31,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:31,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:31,608 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:31,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:31,609 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d7be0b2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:31,609 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:31,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:31,610 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:31,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:31,610 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:31,610 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:31,610 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:31,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:31,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,611 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:31,611 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:31,611 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:31,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:31,613 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:31,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:31,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:31,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:31,613 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:31,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:31,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:31,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:31,615 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:31,618 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:31,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:31,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:31,618 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,618 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:31,618 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:31,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:31,718 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:31,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:31,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:31,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:31,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:31,722 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:31,722 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:31,722 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:31,722 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:31,722 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,722 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:31,724 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:31,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:31,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:31,725 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:31,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:31,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:31,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:31,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:31,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:31,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:31,729 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:31,729 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:31,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:31,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:31,730 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-30 03:56:31,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:31,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:31,731 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:31,731 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:31,731 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:31,731 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:32,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:32,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:32,146 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/06dfb3fe0ca04f9990564fe528b6c87b 2023-03-30 03:56:32,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:32,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148652157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,179 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/f95f8a0310c64c21b073e7c1f90fd128 2023-03-30 03:56:32,208 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/9d96da57589f46269dd0b0f669d5e247 2023-03-30 03:56:32,215 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/06dfb3fe0ca04f9990564fe528b6c87b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/06dfb3fe0ca04f9990564fe528b6c87b 2023-03-30 03:56:32,224 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/06dfb3fe0ca04f9990564fe528b6c87b, entries=150, sequenceid=525, filesize=11.9 K 2023-03-30 03:56:32,225 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/f95f8a0310c64c21b073e7c1f90fd128 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/f95f8a0310c64c21b073e7c1f90fd128 2023-03-30 03:56:32,231 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/f95f8a0310c64c21b073e7c1f90fd128, entries=150, sequenceid=525, filesize=11.9 K 2023-03-30 03:56:32,232 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/9d96da57589f46269dd0b0f669d5e247 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/9d96da57589f46269dd0b0f669d5e247 2023-03-30 03:56:32,245 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/9d96da57589f46269dd0b0f669d5e247, entries=150, sequenceid=525, filesize=11.9 K 2023-03-30 03:56:32,248 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 3ecc6181b1966d72d93cc5e543ef979f in 518ms, sequenceid=525, compaction requested=true 2023-03-30 03:56:32,248 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:32,248 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:32,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:32,249 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,249 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:32,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5caa00a1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:32,251 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:32,251 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,252 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:32,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:32,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:32,252 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,252 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:32,253 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:32,253 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,253 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:32,253 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:32,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:32,256 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:32,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:32,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:32,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:32,256 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:32,258 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,258 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,258 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,258 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,260 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,263 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,263 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,263 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,263 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:32,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,270 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:32,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:32,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,270 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:32,270 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:32,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:32,272 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:56:32,272 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:32,272 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:32,272 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:32,272 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:32,272 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:32,272 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:32,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/0dc815a425a8470bab9c386ab6a4e155 2023-03-30 03:56:32,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/a418e7f29ab54ee09ffb6462dc7c2536 2023-03-30 03:56:32,371 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:32,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:32,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:32,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:32,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:32,381 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:32,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:32,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:32,382 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,382 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:32,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:32,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:32,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148652381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:32,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:32,394 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:32,394 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:32,394 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,394 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:32,394 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:32,394 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,395 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:32,395 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:32,395 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,395 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:32,395 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:32,396 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,396 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:32,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d51a223[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:32,398 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:32,398 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,400 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:32,400 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:32,400 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:32,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:32,401 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:32,401 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,401 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:32,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:32,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:32,405 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:32,405 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:32,405 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:32,405 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:32,405 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:32,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,410 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,418 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,418 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,419 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:32,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:32,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:32,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,419 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:32,419 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:32,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:32,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148652488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,520 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:32,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:32,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:32,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:32,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:32,524 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:32,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:32,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:32,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:32,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:32,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:32,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:32,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:32,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:32,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:32,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:32,527 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,527 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:32,527 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:32,527 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:32,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:32,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,528 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:32,528 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:32,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:32,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79cd0b73[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:32,531 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:32,531 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,532 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:32,532 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:32,532 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:32,533 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,533 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:32,533 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,533 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,533 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,534 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:32,535 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:32,535 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:32,535 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:32,535 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:32,535 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:32,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,537 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,539 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:32,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:32,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,551 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:32,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,551 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:32,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,652 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:32,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:32,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:32,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:32,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:32,654 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:32,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:32,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:32,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,656 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,656 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,656 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,656 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:32,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:32,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:32,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:32,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:32,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:32,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:32,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:32,658 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,658 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:32,658 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:32,658 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:32,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:32,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,659 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:32,659 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:32,659 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,659 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:32,661 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b810a88[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:32,661 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,662 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:32,662 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:32,662 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:32,662 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,662 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:32,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,662 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:32,663 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,663 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:32,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:32,665 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:32,665 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:32,665 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:32,665 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:32,665 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:32,666 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,666 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,666 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,666 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:32,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:32,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:32,669 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:32,668 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,669 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:32,669 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,669 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,669 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,669 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,669 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148652692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/2d21384507d04ad1b21b907fc65043d2 2023-03-30 03:56:32,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/0dc815a425a8470bab9c386ab6a4e155 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0dc815a425a8470bab9c386ab6a4e155 2023-03-30 03:56:32,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0dc815a425a8470bab9c386ab6a4e155, entries=200, sequenceid=537, filesize=14.3 K 2023-03-30 03:56:32,769 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:32,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:32,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/a418e7f29ab54ee09ffb6462dc7c2536 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a418e7f29ab54ee09ffb6462dc7c2536 2023-03-30 03:56:32,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:32,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:32,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:32,772 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:32,772 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,772 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:32,772 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:32,772 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,773 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a418e7f29ab54ee09ffb6462dc7c2536, entries=150, sequenceid=537, filesize=11.9 K 2023-03-30 03:56:32,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/2d21384507d04ad1b21b907fc65043d2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2d21384507d04ad1b21b907fc65043d2 2023-03-30 03:56:32,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,775 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,775 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,775 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:32,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:32,776 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:32,776 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,776 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:32,776 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:32,776 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:32,776 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,776 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:32,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:32,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:32,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:32,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:32,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:32,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,778 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2d21384507d04ad1b21b907fc65043d2, entries=150, sequenceid=537, filesize=11.9 K 2023-03-30 03:56:32,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:32,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4722a451[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:32,780 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:32,780 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3ecc6181b1966d72d93cc5e543ef979f in 510ms, sequenceid=537, compaction requested=true 2023-03-30 03:56:32,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:32,781 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,781 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:56:32,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:32,781 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:32,781 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:32,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:32,781 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:56:32,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:32,781 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:32,782 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,782 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:32,782 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:32,782 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,782 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:32,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,783 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:56:32,783 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66900 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:56:32,783 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:32,783 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,783 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,783 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,783 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/9e128e9ba1c44e41a3449481d13fc965, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/644a65843eca4da4a65ceeb8b8a9a58a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1d231973c1e84b8b9a96d0fb09f9256e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/06dfb3fe0ca04f9990564fe528b6c87b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0dc815a425a8470bab9c386ab6a4e155] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=65.3 K 2023-03-30 03:56:32,783 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6626c58d46e0441aa29577ddd081c5a1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ae4d34fcd9b44cb4824c9a72c6d3e127, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ab16d8d43e645869a6169d6e4cdf218, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/f95f8a0310c64c21b073e7c1f90fd128, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a418e7f29ab54ee09ffb6462dc7c2536] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=60.6 K 2023-03-30 03:56:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,784 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9e128e9ba1c44e41a3449481d13fc965, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680148587820 2023-03-30 03:56:32,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,784 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6626c58d46e0441aa29577ddd081c5a1, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680148587820 2023-03-30 03:56:32,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,785 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 644a65843eca4da4a65ceeb8b8a9a58a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680148589002 2023-03-30 03:56:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,785 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ae4d34fcd9b44cb4824c9a72c6d3e127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680148589002 2023-03-30 03:56:32,785 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1d231973c1e84b8b9a96d0fb09f9256e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680148589666 2023-03-30 03:56:32,785 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6ab16d8d43e645869a6169d6e4cdf218, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680148589666 2023-03-30 03:56:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:32,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:32,786 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:32,786 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:32,786 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:32,786 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:32,786 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:32,786 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting f95f8a0310c64c21b073e7c1f90fd128, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680148590964 2023-03-30 03:56:32,786 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 06dfb3fe0ca04f9990564fe528b6c87b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680148590964 2023-03-30 03:56:32,786 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a418e7f29ab54ee09ffb6462dc7c2536, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680148592115 2023-03-30 03:56:32,787 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 0dc815a425a8470bab9c386ab6a4e155, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680148592115 2023-03-30 03:56:32,787 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,787 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:32,787 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:32,788 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:32,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:32,791 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,791 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,791 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,791 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:32,791 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,791 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:32,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:32,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:32,791 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,792 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,792 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:32,792 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:32,792 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:32,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:32,792 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:32,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:32,804 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#111 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:32,813 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#112 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:32,892 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:32,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:32,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:32,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:32,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:32,894 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:32,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:32,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:32,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,895 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:32,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:32,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:32,896 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:32,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:32,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:32,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:32,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:32,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:32,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:32,897 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:32,897 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:32,897 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:32,897 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:32,897 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:56:32,897 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:32,897 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:32,897 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:32,897 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:32,897 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:32,897 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:32,911 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/b13cc5ec73bb431980d7f83ca9046a84 2023-03-30 03:56:32,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:32,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:33,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148653019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148653120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,225 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/a7695c1177db4a149b709758f43a939a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a7695c1177db4a149b709758f43a939a 2023-03-30 03:56:33,235 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/7ece416f974a4c4089294ff3aa78de14 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/7ece416f974a4c4089294ff3aa78de14 2023-03-30 03:56:33,242 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into 7ece416f974a4c4089294ff3aa78de14(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:33,242 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:33,242 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=11, startTime=1680148592781; duration=0sec 2023-03-30 03:56:33,242 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:33,243 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:56:33,243 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into a7695c1177db4a149b709758f43a939a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:33,243 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:33,243 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=11, startTime=1680148592781; duration=0sec 2023-03-30 03:56:33,243 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:33,244 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:56:33,244 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:33,244 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,245 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c169bfe01098499ebb19da01700e7f8b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7f1cbee8d9c846b298dc4798ce3b3d81, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b978dfc96b984ba487adaa548a9d9149, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/9d96da57589f46269dd0b0f669d5e247, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2d21384507d04ad1b21b907fc65043d2] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=60.6 K 2023-03-30 03:56:33,245 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c169bfe01098499ebb19da01700e7f8b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680148587820 2023-03-30 03:56:33,246 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 7f1cbee8d9c846b298dc4798ce3b3d81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680148589002 2023-03-30 03:56:33,246 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b978dfc96b984ba487adaa548a9d9149, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680148589666 2023-03-30 03:56:33,246 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 9d96da57589f46269dd0b0f669d5e247, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680148590964 2023-03-30 03:56:33,247 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2d21384507d04ad1b21b907fc65043d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680148592115 2023-03-30 03:56:33,261 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#115 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:33,269 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/c0936686042c401490fc15dd91387595 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c0936686042c401490fc15dd91387595 2023-03-30 03:56:33,274 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into c0936686042c401490fc15dd91387595(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:33,274 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:33,274 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=11, startTime=1680148592781; duration=0sec 2023-03-30 03:56:33,275 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:33,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148653323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,327 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/76c4afcff5e4480ebbf0e4815358ad46 2023-03-30 03:56:33,337 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/fbaee1f8780a43f5a65b2107eb6037fd 2023-03-30 03:56:33,342 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/b13cc5ec73bb431980d7f83ca9046a84 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/b13cc5ec73bb431980d7f83ca9046a84 2023-03-30 03:56:33,345 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/b13cc5ec73bb431980d7f83ca9046a84, entries=150, sequenceid=557, filesize=11.9 K 2023-03-30 03:56:33,346 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/76c4afcff5e4480ebbf0e4815358ad46 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/76c4afcff5e4480ebbf0e4815358ad46 2023-03-30 03:56:33,352 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/76c4afcff5e4480ebbf0e4815358ad46, entries=150, sequenceid=557, filesize=11.9 K 2023-03-30 03:56:33,353 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/fbaee1f8780a43f5a65b2107eb6037fd as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fbaee1f8780a43f5a65b2107eb6037fd 2023-03-30 03:56:33,359 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fbaee1f8780a43f5a65b2107eb6037fd, entries=150, sequenceid=557, filesize=11.9 K 2023-03-30 03:56:33,360 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3ecc6181b1966d72d93cc5e543ef979f in 462ms, sequenceid=557, compaction requested=false 2023-03-30 03:56:33,360 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:33,360 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:33,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:33,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:33,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:33,361 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,361 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,361 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,361 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:33,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2702562c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:33,363 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:33,363 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,363 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:33,363 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,363 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:33,364 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,364 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:33,364 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,364 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:33,366 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:33,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:33,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:33,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:33,366 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:33,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,368 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,369 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,370 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:33,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:33,371 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:33,371 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,371 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,371 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:33,371 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,371 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,371 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,371 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,371 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,371 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,371 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:33,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:56:33,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:33,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:33,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:33,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:33,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:33,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:33,460 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/77eee4f311b3405dba84a5ee6502df34 2023-03-30 03:56:33,471 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:33,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:33,474 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:33,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:33,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:33,475 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,475 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,475 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,475 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:33,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:33,477 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,477 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:33,477 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:33,477 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:33,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:33,478 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:33,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:33,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3685572c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:33,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,480 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:33,480 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:33,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:33,480 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:33,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148653478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148653479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,481 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,481 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:33,481 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,481 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:33,483 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:33,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:33,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:33,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:33,483 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:33,484 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,484 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,484 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,484 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,485 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,486 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:33,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:33,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:33,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,487 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:33,487 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:33,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148653582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148653582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,587 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:33,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:33,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:33,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:33,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:33,589 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:33,589 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:33,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:33,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,590 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:33,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:33,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:33,592 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,592 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:33,592 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:33,592 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:33,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:33,593 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,593 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:33,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7513ce3e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:33,595 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:33,595 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,596 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:33,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:33,596 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,596 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:33,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:33,597 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:33,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:33,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:33,599 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:33,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:33,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:33,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:33,599 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:33,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,600 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:33,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:33,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,603 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:33,603 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:33,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:33,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148653627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,703 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:33,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:33,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:33,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:33,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:33,716 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:33,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:33,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:33,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,717 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,717 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,717 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,718 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:33,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:33,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:33,718 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:33,718 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,718 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:33,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:33,721 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:33,721 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:33,721 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:33,721 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:33,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:33,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:33,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:33,722 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,722 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,722 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,722 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2741ece1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:33,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:33,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,724 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:33,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:33,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:33,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:33,725 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,725 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,725 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,725 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:33,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:33,727 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:33,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:33,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:33,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:33,727 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:33,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,729 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,730 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,730 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,730 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,730 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,730 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:33,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:33,730 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,731 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:33,731 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:33,731 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,731 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,731 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:33,731 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148653783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:33,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148653784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,831 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:33,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:33,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:33,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:33,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:33,834 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:33,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:33,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:33,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,835 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,835 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,835 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,835 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:33,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:33,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:33,836 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,836 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:33,836 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:33,836 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:33,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:33,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,837 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,837 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,837 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c76d13[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:33,838 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:33,839 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,839 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,839 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:33,839 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:33,839 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,839 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:33,839 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:33,840 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,840 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,840 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,840 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:33,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:33,842 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:33,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:33,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:33,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:33,842 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:33,843 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,843 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,843 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,845 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,846 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,846 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,846 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:33,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:33,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:33,846 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:33,847 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:33,846 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,846 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,846 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,847 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,847 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,847 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,847 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/a0015f3ec31940db9fd2aa76a03ee486 2023-03-30 03:56:33,947 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:33,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:33,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:33,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:33,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:33,949 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:33,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:33,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:33,950 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,950 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,950 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,950 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:33,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:33,952 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,952 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:33,952 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:33,952 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:33,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,953 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,953 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,953 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee76450[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:33,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:33,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,955 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:33,956 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:33,958 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:33,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:33,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:33,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:33,958 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:33,959 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,959 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:33,959 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:33,959 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:33,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:33,962 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:33,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:33,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:33,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:33,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:33,962 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:33,963 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:33,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:34,063 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:34,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:34,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:34,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:34,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:34,066 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:34,066 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:34,066 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:34,066 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:34,066 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,066 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,067 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:34,067 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:34,067 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,067 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:34,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:34,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:34,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:34,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:34,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:34,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:34,068 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,068 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:34,069 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:34,069 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:34,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:34,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:34,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:34,070 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,070 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:34,070 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,070 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:34,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:34,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:34,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656e398c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:34,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,072 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:34,072 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:34,072 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:34,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:34,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,072 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:34,072 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:34,073 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:34,074 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:34,074 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:34,074 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:34,074 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:34,074 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:34,075 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,075 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,075 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:34,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,077 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:34,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:34,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:34,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:34,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:34,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:34,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:34,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:34,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:34,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,079 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:34,079 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:34,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:34,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37658 deadline: 1680148654086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:34,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37688 deadline: 1680148654087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:34,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37654 deadline: 1680148654130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:34,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37670 deadline: 1680148654144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,147 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18220 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:34,179 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:34,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:34,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:34,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:34,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:34,181 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:34,181 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:34,181 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:34,181 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:34,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:34,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:34,183 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,183 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:34,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:34,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:34,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:34,184 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,184 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:34,184 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:34,184 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:34,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:34,185 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:34,186 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,186 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d60e724[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:34,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:34,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,188 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,188 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,188 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:34,188 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:34,188 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:34,188 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:34,189 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:34,191 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:34,191 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:34,191 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:34,191 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:34,191 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:34,192 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,192 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:34,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,192 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:34,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,194 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,196 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:34,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:34,196 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:34,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:34,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:34,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:34,196 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,196 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,196 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,196 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,196 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:34,196 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:34,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:34,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:34,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/5b7690ba4c6b471697fd72aaea520280 2023-03-30 03:56:34,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/77eee4f311b3405dba84a5ee6502df34 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/77eee4f311b3405dba84a5ee6502df34 2023-03-30 03:56:34,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/77eee4f311b3405dba84a5ee6502df34, entries=200, sequenceid=573, filesize=14.3 K 2023-03-30 03:56:34,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/a0015f3ec31940db9fd2aa76a03ee486 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a0015f3ec31940db9fd2aa76a03ee486 2023-03-30 03:56:34,296 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:34,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:34,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:34,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a0015f3ec31940db9fd2aa76a03ee486, entries=150, sequenceid=573, filesize=11.9 K 2023-03-30 03:56:34,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:34,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:34,299 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:34,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:34,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:34,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:34,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/5b7690ba4c6b471697fd72aaea520280 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/5b7690ba4c6b471697fd72aaea520280 2023-03-30 03:56:34,300 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:34,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:34,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:34,301 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:34,301 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:34,301 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:34,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:34,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:34,302 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,302 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,302 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:34,302 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:34,302 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,302 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,302 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:34,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:34,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:34,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:34,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:34,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:34,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:34,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:34,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/5b7690ba4c6b471697fd72aaea520280, entries=150, sequenceid=573, filesize=11.9 K 2023-03-30 03:56:34,305 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:34,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e6a46e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:34,305 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,305 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3ecc6181b1966d72d93cc5e543ef979f in 854ms, sequenceid=573, compaction requested=true 2023-03-30 03:56:34,306 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:34,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:34,306 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:34,306 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:34,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:34,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:34,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:34,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:34,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:34,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:34,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:34,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:34,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:34,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:34,308 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:34,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:34,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:34,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:34,308 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:34,310 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40258 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:34,310 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/A is initiating minor compaction (all files) 2023-03-30 03:56:34,310 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:34,310 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/A in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,310 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/B is initiating minor compaction (all files) 2023-03-30 03:56:34,310 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/B in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,310 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a7695c1177db4a149b709758f43a939a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/b13cc5ec73bb431980d7f83ca9046a84, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/77eee4f311b3405dba84a5ee6502df34] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=39.3 K 2023-03-30 03:56:34,310 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/7ece416f974a4c4089294ff3aa78de14, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/76c4afcff5e4480ebbf0e4815358ad46, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a0015f3ec31940db9fd2aa76a03ee486] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=36.9 K 2023-03-30 03:56:34,310 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting a7695c1177db4a149b709758f43a939a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680148592115 2023-03-30 03:56:34,310 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 7ece416f974a4c4089294ff3aa78de14, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680148592115 2023-03-30 03:56:34,311 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting b13cc5ec73bb431980d7f83ca9046a84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1680148592350 2023-03-30 03:56:34,311 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 76c4afcff5e4480ebbf0e4815358ad46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1680148592350 2023-03-30 03:56:34,311 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,311 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:34,311 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 77eee4f311b3405dba84a5ee6502df34, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680148593001 2023-03-30 03:56:34,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,311 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:34,312 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a0015f3ec31940db9fd2aa76a03ee486, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680148593001 2023-03-30 03:56:34,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:34,312 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:34,314 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,315 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:34,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:34,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:34,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:34,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:34,316 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:34,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,316 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:34,316 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:34,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:34,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:34,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:34,319 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#B#compaction#120 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:34,321 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#A#compaction#121 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:34,340 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/fd32937cbcaf4515bced9cd6768394fd as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fd32937cbcaf4515bced9cd6768394fd 2023-03-30 03:56:34,340 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/6700a24c617b47cba0991527cf289c11 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/6700a24c617b47cba0991527cf289c11 2023-03-30 03:56:34,345 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/A of 3ecc6181b1966d72d93cc5e543ef979f into 6700a24c617b47cba0991527cf289c11(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:34,345 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:34,345 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/A, priority=13, startTime=1680148594306; duration=0sec 2023-03-30 03:56:34,345 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:34,345 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:34,345 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/B of 3ecc6181b1966d72d93cc5e543ef979f into fd32937cbcaf4515bced9cd6768394fd(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:34,345 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:34,345 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/B, priority=13, startTime=1680148594306; duration=0sec 2023-03-30 03:56:34,345 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:34,346 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:34,346 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 3ecc6181b1966d72d93cc5e543ef979f/C is initiating minor compaction (all files) 2023-03-30 03:56:34,346 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3ecc6181b1966d72d93cc5e543ef979f/C in TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,346 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c0936686042c401490fc15dd91387595, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fbaee1f8780a43f5a65b2107eb6037fd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/5b7690ba4c6b471697fd72aaea520280] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp, totalSize=36.9 K 2023-03-30 03:56:34,347 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting c0936686042c401490fc15dd91387595, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680148592115 2023-03-30 03:56:34,347 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting fbaee1f8780a43f5a65b2107eb6037fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1680148592350 2023-03-30 03:56:34,348 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5b7690ba4c6b471697fd72aaea520280, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680148593001 2023-03-30 03:56:34,353 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3ecc6181b1966d72d93cc5e543ef979f#C#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:34,416 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:34,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:34,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:34,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:34,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:34,419 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:34,419 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:34,419 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:34,419 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:34,419 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,420 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:34,420 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:34,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:34,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:34,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:34,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:34,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:34,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:34,421 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:34,421 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:34,421 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:34,422 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:34,422 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:34,422 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:34,422 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. started... 2023-03-30 03:56:34,422 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 03:56:34,422 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:34,422 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:34,422 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:34,422 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:34,422 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:34,422 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:34,429 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/3b3438c0d8fe44dba146e1c75ae78338 2023-03-30 03:56:34,438 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5fe050a3 to 127.0.0.1:58033 2023-03-30 03:56:34,438 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:34,438 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x655398b8 to 127.0.0.1:58033 2023-03-30 03:56:34,438 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:34,442 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13112432 to 127.0.0.1:58033 2023-03-30 03:56:34,442 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:34,444 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x776686ab to 127.0.0.1:58033 2023-03-30 03:56:34,444 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:34,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:34,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. as already flushing 2023-03-30 03:56:34,595 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20e4e4f3 to 127.0.0.1:58033 2023-03-30 03:56:34,595 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f537ba4 to 127.0.0.1:58033 2023-03-30 03:56:34,595 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:34,595 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:34,773 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/b95896f5e83d43dda0fe4d61881838a7 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b95896f5e83d43dda0fe4d61881838a7 2023-03-30 03:56:34,780 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3ecc6181b1966d72d93cc5e543ef979f/C of 3ecc6181b1966d72d93cc5e543ef979f into b95896f5e83d43dda0fe4d61881838a7(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:34,780 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:34,780 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f., storeName=3ecc6181b1966d72d93cc5e543ef979f/C, priority=13, startTime=1680148594306; duration=0sec 2023-03-30 03:56:34,780 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:34,852 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/3736687828b64ee9b1463596252fa4e6 2023-03-30 03:56:35,139 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6362e5a0 to 127.0.0.1:58033 2023-03-30 03:56:35,139 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:35,273 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/c2b2021476334b2ca056aab5c210e837 2023-03-30 03:56:35,283 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/3b3438c0d8fe44dba146e1c75ae78338 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/3b3438c0d8fe44dba146e1c75ae78338 2023-03-30 03:56:35,289 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/3b3438c0d8fe44dba146e1c75ae78338, entries=150, sequenceid=594, filesize=11.9 K 2023-03-30 03:56:35,291 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/3736687828b64ee9b1463596252fa4e6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3736687828b64ee9b1463596252fa4e6 2023-03-30 03:56:35,298 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3736687828b64ee9b1463596252fa4e6, entries=150, sequenceid=594, filesize=11.9 K 2023-03-30 03:56:35,299 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/c2b2021476334b2ca056aab5c210e837 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c2b2021476334b2ca056aab5c210e837 2023-03-30 03:56:35,304 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c2b2021476334b2ca056aab5c210e837, entries=150, sequenceid=594, filesize=11.9 K 2023-03-30 03:56:35,304 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=20.13 KB/20610 for 3ecc6181b1966d72d93cc5e543ef979f in 882ms, sequenceid=594, compaction requested=false 2023-03-30 03:56:35,304 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:35,305 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:35,305 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:35,305 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:35,305 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,305 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:35,305 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:35,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:35,306 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:35,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:35,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51169946[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:35,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:35,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:35,308 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:35,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:35,308 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:35,308 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:35,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:35,308 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:35,309 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:35,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:35,310 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:35,310 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:35,310 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:35,310 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:35,310 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:35,311 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:35,311 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:35,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:35,311 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:35,312 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:35,312 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:35,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:35,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:35,314 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:35,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:35,314 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:35,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:35,314 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:35,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:35,414 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:35,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:38,719 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x046b5e40 to 127.0.0.1:58033 2023-03-30 03:56:38,719 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:44,238 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x142c8aea to 127.0.0.1:58033 2023-03-30 03:56:44,239 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:44,239 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 03:56:44,239 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 98 2023-03-30 03:56:44,240 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 7 2023-03-30 03:56:44,240 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 248 2023-03-30 03:56:44,240 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-03-30 03:56:44,241 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-03-30 03:56:44,241 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 03:56:44,241 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4346 2023-03-30 03:56:44,242 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4323 2023-03-30 03:56:44,242 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 03:56:44,242 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 1938 2023-03-30 03:56:44,242 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 5804 rows 2023-03-30 03:56:44,242 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 1946 2023-03-30 03:56:44,242 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 5837 rows 2023-03-30 03:56:44,242 INFO [Listener at localhost.localdomain/37215] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 03:56:44,242 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x458f9a9e to 127.0.0.1:58033 2023-03-30 03:56:44,242 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:56:44,246 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 03:56:44,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-30 03:56:44,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:44,262 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148604262"}]},"ts":"1680148604262"} 2023-03-30 03:56:44,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 03:56:44,264 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 03:56:44,287 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 03:56:44,289 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ecc6181b1966d72d93cc5e543ef979f, UNASSIGN}] 2023-03-30 03:56:44,290 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ecc6181b1966d72d93cc5e543ef979f, UNASSIGN 2023-03-30 03:56:44,291 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3ecc6181b1966d72d93cc5e543ef979f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:44,291 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148604291"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148604291"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148604291"}]},"ts":"1680148604291"} 2023-03-30 03:56:44,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:56:44,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 03:56:44,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:44,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 3ecc6181b1966d72d93cc5e543ef979f, disabling compactions & flushes 2023-03-30 03:56:44,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:44,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:44,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. after waiting 0 ms 2023-03-30 03:56:44,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:44,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 3ecc6181b1966d72d93cc5e543ef979f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 03:56:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=A 2023-03-30 03:56:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=B 2023-03-30 03:56:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3ecc6181b1966d72d93cc5e543ef979f, store=C 2023-03-30 03:56:44,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:44,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 03:56:44,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 03:56:44,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/e87d58d67f174e31afb7891ae7b16922 2023-03-30 03:56:44,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/079c808e26874461858ec054082c9866 2023-03-30 03:56:45,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/c08918ea03cd4bb19e3d0025f8cf0786 2023-03-30 03:56:45,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/A/e87d58d67f174e31afb7891ae7b16922 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/e87d58d67f174e31afb7891ae7b16922 2023-03-30 03:56:45,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/e87d58d67f174e31afb7891ae7b16922, entries=100, sequenceid=603, filesize=9.5 K 2023-03-30 03:56:45,323 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/B/079c808e26874461858ec054082c9866 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/079c808e26874461858ec054082c9866 2023-03-30 03:56:45,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/079c808e26874461858ec054082c9866, entries=100, sequenceid=603, filesize=9.5 K 2023-03-30 03:56:45,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/.tmp/C/c08918ea03cd4bb19e3d0025f8cf0786 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c08918ea03cd4bb19e3d0025f8cf0786 2023-03-30 03:56:45,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c08918ea03cd4bb19e3d0025f8cf0786, entries=100, sequenceid=603, filesize=9.5 K 2023-03-30 03:56:45,335 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 3ecc6181b1966d72d93cc5e543ef979f in 882ms, sequenceid=603, compaction requested=true 2023-03-30 03:56:45,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/11eb7f8c1ee340d3a93a94034aa7b93d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d13d757ae1444c1bb3e79aa76a660ed4, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/264772cfba1c4ec6b7ae7bbc2ebcc33a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7912677c614e46bcbf22177bde1991e9, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/64dd662831be4b81b94c970510615601, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/98e140b9a99f4754a7ade4b806d97d23, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d3cc4b826f34467d9e0a8437980a5039, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/bf7bc7b58f5040a79cd75a571b169afd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/68e8f009e41a42d99bb0440f5d526df6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5f95bdd6dc7041e0b65df8137cd18188, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/13922bdaca7b4f26a45054a67b874b6c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/26b0e9c65e25446897963ed56e4a6565, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d5dfed8f977d4ab5adfe053b3bd79569, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1c531974ffbd4829ac756106236f6894, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7861088115f74d6cb96faae801c4f35c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/51cd6a92c9354091b9967fe46bd122ee, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/74abe3a1a0a64c009275ef6a1381b6d9, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/c9e5bb577a6d4fb49545a9238369d2e6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/aa476678f56143dfaaf6c9972b058405, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec159cac0ec340c58e9b89323b6fe4eb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/2b7e48556973481d81f8b6dee44379bc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/18a3a143efdf48189d30de1ae8c06445, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/6e69cafda9f24fa4992cc3937bbec8fb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/03ef8ee055e64686bf7a6e97524c85ba, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/772275cc2b7946839aa06b2b85bd3c47, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0fd2c11d904e45ef9c51d3172125e78e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a29f237d45f548cd8eb6613e26ce0410, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/4f409dfba5024dd8ba9cb33f7ae77852, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5ea960a4861442fb9b14228de60da4a9, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/705c8032cbc14a119d5342814816b3ed, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0b8033a136914a9a8ab5b23f7be5960c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec76b48b513743c4b154cd813c0028e3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/9e128e9ba1c44e41a3449481d13fc965, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/644a65843eca4da4a65ceeb8b8a9a58a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1d231973c1e84b8b9a96d0fb09f9256e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/06dfb3fe0ca04f9990564fe528b6c87b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0dc815a425a8470bab9c386ab6a4e155, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a7695c1177db4a149b709758f43a939a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/b13cc5ec73bb431980d7f83ca9046a84, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/77eee4f311b3405dba84a5ee6502df34] to archive 2023-03-30 03:56:45,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:56:45,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/11eb7f8c1ee340d3a93a94034aa7b93d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/11eb7f8c1ee340d3a93a94034aa7b93d 2023-03-30 03:56:45,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d13d757ae1444c1bb3e79aa76a660ed4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d13d757ae1444c1bb3e79aa76a660ed4 2023-03-30 03:56:45,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/264772cfba1c4ec6b7ae7bbc2ebcc33a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/264772cfba1c4ec6b7ae7bbc2ebcc33a 2023-03-30 03:56:45,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7912677c614e46bcbf22177bde1991e9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7912677c614e46bcbf22177bde1991e9 2023-03-30 03:56:45,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/64dd662831be4b81b94c970510615601 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/64dd662831be4b81b94c970510615601 2023-03-30 03:56:45,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/98e140b9a99f4754a7ade4b806d97d23 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/98e140b9a99f4754a7ade4b806d97d23 2023-03-30 03:56:45,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d3cc4b826f34467d9e0a8437980a5039 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d3cc4b826f34467d9e0a8437980a5039 2023-03-30 03:56:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 03:56:45,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/bf7bc7b58f5040a79cd75a571b169afd to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/bf7bc7b58f5040a79cd75a571b169afd 2023-03-30 03:56:45,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/68e8f009e41a42d99bb0440f5d526df6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/68e8f009e41a42d99bb0440f5d526df6 2023-03-30 03:56:45,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5f95bdd6dc7041e0b65df8137cd18188 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5f95bdd6dc7041e0b65df8137cd18188 2023-03-30 03:56:45,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/13922bdaca7b4f26a45054a67b874b6c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/13922bdaca7b4f26a45054a67b874b6c 2023-03-30 03:56:45,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/26b0e9c65e25446897963ed56e4a6565 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/26b0e9c65e25446897963ed56e4a6565 2023-03-30 03:56:45,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d5dfed8f977d4ab5adfe053b3bd79569 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/d5dfed8f977d4ab5adfe053b3bd79569 2023-03-30 03:56:45,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1c531974ffbd4829ac756106236f6894 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1c531974ffbd4829ac756106236f6894 2023-03-30 03:56:45,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7861088115f74d6cb96faae801c4f35c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/7861088115f74d6cb96faae801c4f35c 2023-03-30 03:56:45,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/51cd6a92c9354091b9967fe46bd122ee to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/51cd6a92c9354091b9967fe46bd122ee 2023-03-30 03:56:45,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/74abe3a1a0a64c009275ef6a1381b6d9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/74abe3a1a0a64c009275ef6a1381b6d9 2023-03-30 03:56:45,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/c9e5bb577a6d4fb49545a9238369d2e6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/c9e5bb577a6d4fb49545a9238369d2e6 2023-03-30 03:56:45,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/aa476678f56143dfaaf6c9972b058405 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/aa476678f56143dfaaf6c9972b058405 2023-03-30 03:56:45,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec159cac0ec340c58e9b89323b6fe4eb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec159cac0ec340c58e9b89323b6fe4eb 2023-03-30 03:56:45,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/2b7e48556973481d81f8b6dee44379bc to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/2b7e48556973481d81f8b6dee44379bc 2023-03-30 03:56:45,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/18a3a143efdf48189d30de1ae8c06445 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/18a3a143efdf48189d30de1ae8c06445 2023-03-30 03:56:45,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/6e69cafda9f24fa4992cc3937bbec8fb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/6e69cafda9f24fa4992cc3937bbec8fb 2023-03-30 03:56:45,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/03ef8ee055e64686bf7a6e97524c85ba to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/03ef8ee055e64686bf7a6e97524c85ba 2023-03-30 03:56:45,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/772275cc2b7946839aa06b2b85bd3c47 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/772275cc2b7946839aa06b2b85bd3c47 2023-03-30 03:56:45,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0fd2c11d904e45ef9c51d3172125e78e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0fd2c11d904e45ef9c51d3172125e78e 2023-03-30 03:56:45,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a29f237d45f548cd8eb6613e26ce0410 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a29f237d45f548cd8eb6613e26ce0410 2023-03-30 03:56:45,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/4f409dfba5024dd8ba9cb33f7ae77852 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/4f409dfba5024dd8ba9cb33f7ae77852 2023-03-30 03:56:45,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5ea960a4861442fb9b14228de60da4a9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/5ea960a4861442fb9b14228de60da4a9 2023-03-30 03:56:45,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/705c8032cbc14a119d5342814816b3ed to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/705c8032cbc14a119d5342814816b3ed 2023-03-30 03:56:45,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0b8033a136914a9a8ab5b23f7be5960c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0b8033a136914a9a8ab5b23f7be5960c 2023-03-30 03:56:45,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec76b48b513743c4b154cd813c0028e3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/ec76b48b513743c4b154cd813c0028e3 2023-03-30 03:56:45,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/9e128e9ba1c44e41a3449481d13fc965 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/9e128e9ba1c44e41a3449481d13fc965 2023-03-30 03:56:45,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/644a65843eca4da4a65ceeb8b8a9a58a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/644a65843eca4da4a65ceeb8b8a9a58a 2023-03-30 03:56:45,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1d231973c1e84b8b9a96d0fb09f9256e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/1d231973c1e84b8b9a96d0fb09f9256e 2023-03-30 03:56:45,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/06dfb3fe0ca04f9990564fe528b6c87b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/06dfb3fe0ca04f9990564fe528b6c87b 2023-03-30 03:56:45,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0dc815a425a8470bab9c386ab6a4e155 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/0dc815a425a8470bab9c386ab6a4e155 2023-03-30 03:56:45,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a7695c1177db4a149b709758f43a939a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/a7695c1177db4a149b709758f43a939a 2023-03-30 03:56:45,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/b13cc5ec73bb431980d7f83ca9046a84 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/b13cc5ec73bb431980d7f83ca9046a84 2023-03-30 03:56:45,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/77eee4f311b3405dba84a5ee6502df34 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/77eee4f311b3405dba84a5ee6502df34 2023-03-30 03:56:45,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fc1ff4cba8ae4f3f96aaa20a9f3b4cf3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ebb0a72bad34ebcbffb9b6da4018922, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8f0ee0b778d64f85ba9c3772f6102b83, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ce8e80dba5bd4e35979baff6dcf97c18, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/144689632f964c49b5c149c91c3506f6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2f1835b3650445a99f4a392b589def42, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/c81011d549254002a2cec535c43469b6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/b7b84a34fa774a65b54544c2833cb87b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3fdf145905e449c19d5298e91e8679e0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/efe17035b6e7491aaf6462edf6f454db, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ba7fc0292eac42bcad19e0c382ddc397, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2aa0e6f058ab4a9db0a955cf8eca845c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/04058b3fe2c143088e6034d52b6cfa04, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e1d10e14f9754137abee87ce1f325e9d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/55c7b64677474e39ad2c5f3985028a0c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6dc52327532e44688eb3a42029a2be33, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6b1ee89a0a7b442b86b9146462630cd6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/97ed502d8aef4d229937f9f19c16e2b6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d3daea26028649be89e9b3134df757c1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/9dfd58958a954b4bb183236033b7faed, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/73397c9a2a714f0a93e618d7ceeb0423, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e4de537eeff447f080e45daaf0b85cc8, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0d75ae1610c49b58bedf2505d06702d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a2eda347cdd8416ca8748481bf1d213d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/03149bf0b8ec4bb783c4772742e6e664, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ac5ecd6ef7e642a7b4b02773a3c77c78, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/477fab27aeb44408932c0c8839cbc35a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/531e5331e00841af90ddebbbb0efeb30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fd8e21f31de6420d945246e631d26f92, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/feba0d7988ec439aaffad3bdb6ca7945, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0c2f1377638455b91f0a99834b66ed0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6626c58d46e0441aa29577ddd081c5a1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8dca0777663c4d18b5907f77e9c657de, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ae4d34fcd9b44cb4824c9a72c6d3e127, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ab16d8d43e645869a6169d6e4cdf218, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/f95f8a0310c64c21b073e7c1f90fd128, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/7ece416f974a4c4089294ff3aa78de14, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a418e7f29ab54ee09ffb6462dc7c2536, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/76c4afcff5e4480ebbf0e4815358ad46, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a0015f3ec31940db9fd2aa76a03ee486] to archive 2023-03-30 03:56:45,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:56:45,431 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fc1ff4cba8ae4f3f96aaa20a9f3b4cf3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fc1ff4cba8ae4f3f96aaa20a9f3b4cf3 2023-03-30 03:56:45,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ebb0a72bad34ebcbffb9b6da4018922 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ebb0a72bad34ebcbffb9b6da4018922 2023-03-30 03:56:45,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8f0ee0b778d64f85ba9c3772f6102b83 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8f0ee0b778d64f85ba9c3772f6102b83 2023-03-30 03:56:45,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ce8e80dba5bd4e35979baff6dcf97c18 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ce8e80dba5bd4e35979baff6dcf97c18 2023-03-30 03:56:45,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/144689632f964c49b5c149c91c3506f6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/144689632f964c49b5c149c91c3506f6 2023-03-30 03:56:45,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2f1835b3650445a99f4a392b589def42 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2f1835b3650445a99f4a392b589def42 2023-03-30 03:56:45,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/c81011d549254002a2cec535c43469b6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/c81011d549254002a2cec535c43469b6 2023-03-30 03:56:45,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/b7b84a34fa774a65b54544c2833cb87b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/b7b84a34fa774a65b54544c2833cb87b 2023-03-30 03:56:45,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3fdf145905e449c19d5298e91e8679e0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3fdf145905e449c19d5298e91e8679e0 2023-03-30 03:56:45,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/efe17035b6e7491aaf6462edf6f454db to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/efe17035b6e7491aaf6462edf6f454db 2023-03-30 03:56:45,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ba7fc0292eac42bcad19e0c382ddc397 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ba7fc0292eac42bcad19e0c382ddc397 2023-03-30 03:56:45,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2aa0e6f058ab4a9db0a955cf8eca845c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/2aa0e6f058ab4a9db0a955cf8eca845c 2023-03-30 03:56:45,444 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/04058b3fe2c143088e6034d52b6cfa04 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/04058b3fe2c143088e6034d52b6cfa04 2023-03-30 03:56:45,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e1d10e14f9754137abee87ce1f325e9d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e1d10e14f9754137abee87ce1f325e9d 2023-03-30 03:56:45,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/55c7b64677474e39ad2c5f3985028a0c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/55c7b64677474e39ad2c5f3985028a0c 2023-03-30 03:56:45,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6dc52327532e44688eb3a42029a2be33 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6dc52327532e44688eb3a42029a2be33 2023-03-30 03:56:45,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6b1ee89a0a7b442b86b9146462630cd6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6b1ee89a0a7b442b86b9146462630cd6 2023-03-30 03:56:45,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/97ed502d8aef4d229937f9f19c16e2b6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/97ed502d8aef4d229937f9f19c16e2b6 2023-03-30 03:56:45,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d3daea26028649be89e9b3134df757c1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d3daea26028649be89e9b3134df757c1 2023-03-30 03:56:45,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/9dfd58958a954b4bb183236033b7faed to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/9dfd58958a954b4bb183236033b7faed 2023-03-30 03:56:45,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/73397c9a2a714f0a93e618d7ceeb0423 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/73397c9a2a714f0a93e618d7ceeb0423 2023-03-30 03:56:45,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e4de537eeff447f080e45daaf0b85cc8 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/e4de537eeff447f080e45daaf0b85cc8 2023-03-30 03:56:45,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0d75ae1610c49b58bedf2505d06702d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0d75ae1610c49b58bedf2505d06702d 2023-03-30 03:56:45,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a2eda347cdd8416ca8748481bf1d213d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a2eda347cdd8416ca8748481bf1d213d 2023-03-30 03:56:45,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/03149bf0b8ec4bb783c4772742e6e664 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/03149bf0b8ec4bb783c4772742e6e664 2023-03-30 03:56:45,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ac5ecd6ef7e642a7b4b02773a3c77c78 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ac5ecd6ef7e642a7b4b02773a3c77c78 2023-03-30 03:56:45,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/477fab27aeb44408932c0c8839cbc35a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/477fab27aeb44408932c0c8839cbc35a 2023-03-30 03:56:45,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/531e5331e00841af90ddebbbb0efeb30 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/531e5331e00841af90ddebbbb0efeb30 2023-03-30 03:56:45,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fd8e21f31de6420d945246e631d26f92 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fd8e21f31de6420d945246e631d26f92 2023-03-30 03:56:45,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/feba0d7988ec439aaffad3bdb6ca7945 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/feba0d7988ec439aaffad3bdb6ca7945 2023-03-30 03:56:45,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0c2f1377638455b91f0a99834b66ed0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/d0c2f1377638455b91f0a99834b66ed0 2023-03-30 03:56:45,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6626c58d46e0441aa29577ddd081c5a1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6626c58d46e0441aa29577ddd081c5a1 2023-03-30 03:56:45,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8dca0777663c4d18b5907f77e9c657de to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/8dca0777663c4d18b5907f77e9c657de 2023-03-30 03:56:45,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ae4d34fcd9b44cb4824c9a72c6d3e127 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/ae4d34fcd9b44cb4824c9a72c6d3e127 2023-03-30 03:56:45,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ab16d8d43e645869a6169d6e4cdf218 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/6ab16d8d43e645869a6169d6e4cdf218 2023-03-30 03:56:45,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/f95f8a0310c64c21b073e7c1f90fd128 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/f95f8a0310c64c21b073e7c1f90fd128 2023-03-30 03:56:45,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/7ece416f974a4c4089294ff3aa78de14 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/7ece416f974a4c4089294ff3aa78de14 2023-03-30 03:56:45,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a418e7f29ab54ee09ffb6462dc7c2536 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a418e7f29ab54ee09ffb6462dc7c2536 2023-03-30 03:56:45,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/76c4afcff5e4480ebbf0e4815358ad46 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/76c4afcff5e4480ebbf0e4815358ad46 2023-03-30 03:56:45,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a0015f3ec31940db9fd2aa76a03ee486 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/a0015f3ec31940db9fd2aa76a03ee486 2023-03-30 03:56:45,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2ec8035aacc44ba190936bdf3806945f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7ca599eddef744bc80ec474584abe48b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/873a1330f82146fa80e5a0339702d110, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3ca54b7bb9a04fb8908a7bbab2ff5ae4, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d7cb3efc92ab4c48b1bfb28354c381c7, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/8dd3137f3a0b437684fd4e1fb8bb1e6a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d34cf8cf2c0a43a4b464d9222316e3ba, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/aeffcd57619d4a4da0ede7f5d35a2831, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c7ba2d54bafe48f88d133401d2d580f1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/92556dd6cf7b4c4ca325cd9d9963ab2a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ae8670823e6342978de61f25f808441a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b590e1e07c4c4049a91389f2f95cdfd1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/38139219f31d4e22b21f1467c34af3ae, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/173d4ee5bae64920b2ff326f7351f0f2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3bcca592da2c41e1ab8e8fbd5a21e715, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/79ba1dbd6a79414390c067bfbda29349, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ee533cb1149b4520816f24e3373833e5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b488791e97c942f4ad76c7c41058abe5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/cc8daf17389e4ba19d47ac7264ebc3cc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ce116fe7313f40f59890628ac2886305, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/1a8b4897ca4a4c398f4558e15197bc9f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fe6c359553e34eb2ba62464bd16e456c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/851d6a2378c847c8a2da3adb09cd79cb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/44942974ed5f4346879a164e730bfc58, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/85e6adba8f3d4c1aa0d0a280e926408a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/f958a8cc0c7844088c775a9d91af77cd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b49c3d9576544ee2986fb8835af69703, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3c78920eccf04dc4ba0a90d215d9d48e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/6fb67444cd0f4925b0c3006a9519607b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d83bd16d7f58468490ef07f9c90d02ae, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/181e7a7b223a4eee988a77bb21f152e7, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c169bfe01098499ebb19da01700e7f8b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fa2f3799c4db4fc5b2d5c6bdb654f525, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7f1cbee8d9c846b298dc4798ce3b3d81, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b978dfc96b984ba487adaa548a9d9149, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/9d96da57589f46269dd0b0f669d5e247, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c0936686042c401490fc15dd91387595, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2d21384507d04ad1b21b907fc65043d2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fbaee1f8780a43f5a65b2107eb6037fd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/5b7690ba4c6b471697fd72aaea520280] to archive 2023-03-30 03:56:45,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:56:45,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2ec8035aacc44ba190936bdf3806945f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2ec8035aacc44ba190936bdf3806945f 2023-03-30 03:56:45,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7ca599eddef744bc80ec474584abe48b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7ca599eddef744bc80ec474584abe48b 2023-03-30 03:56:45,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/873a1330f82146fa80e5a0339702d110 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/873a1330f82146fa80e5a0339702d110 2023-03-30 03:56:45,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3ca54b7bb9a04fb8908a7bbab2ff5ae4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3ca54b7bb9a04fb8908a7bbab2ff5ae4 2023-03-30 03:56:45,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d7cb3efc92ab4c48b1bfb28354c381c7 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d7cb3efc92ab4c48b1bfb28354c381c7 2023-03-30 03:56:45,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/8dd3137f3a0b437684fd4e1fb8bb1e6a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/8dd3137f3a0b437684fd4e1fb8bb1e6a 2023-03-30 03:56:45,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d34cf8cf2c0a43a4b464d9222316e3ba to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d34cf8cf2c0a43a4b464d9222316e3ba 2023-03-30 03:56:45,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/aeffcd57619d4a4da0ede7f5d35a2831 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/aeffcd57619d4a4da0ede7f5d35a2831 2023-03-30 03:56:45,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c7ba2d54bafe48f88d133401d2d580f1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c7ba2d54bafe48f88d133401d2d580f1 2023-03-30 03:56:45,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/92556dd6cf7b4c4ca325cd9d9963ab2a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/92556dd6cf7b4c4ca325cd9d9963ab2a 2023-03-30 03:56:45,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ae8670823e6342978de61f25f808441a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ae8670823e6342978de61f25f808441a 2023-03-30 03:56:45,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b590e1e07c4c4049a91389f2f95cdfd1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b590e1e07c4c4049a91389f2f95cdfd1 2023-03-30 03:56:45,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/38139219f31d4e22b21f1467c34af3ae to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/38139219f31d4e22b21f1467c34af3ae 2023-03-30 03:56:45,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/173d4ee5bae64920b2ff326f7351f0f2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/173d4ee5bae64920b2ff326f7351f0f2 2023-03-30 03:56:45,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3bcca592da2c41e1ab8e8fbd5a21e715 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3bcca592da2c41e1ab8e8fbd5a21e715 2023-03-30 03:56:45,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/79ba1dbd6a79414390c067bfbda29349 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/79ba1dbd6a79414390c067bfbda29349 2023-03-30 03:56:45,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ee533cb1149b4520816f24e3373833e5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ee533cb1149b4520816f24e3373833e5 2023-03-30 03:56:45,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b488791e97c942f4ad76c7c41058abe5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b488791e97c942f4ad76c7c41058abe5 2023-03-30 03:56:45,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/cc8daf17389e4ba19d47ac7264ebc3cc to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/cc8daf17389e4ba19d47ac7264ebc3cc 2023-03-30 03:56:45,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ce116fe7313f40f59890628ac2886305 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/ce116fe7313f40f59890628ac2886305 2023-03-30 03:56:45,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/1a8b4897ca4a4c398f4558e15197bc9f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/1a8b4897ca4a4c398f4558e15197bc9f 2023-03-30 03:56:45,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fe6c359553e34eb2ba62464bd16e456c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fe6c359553e34eb2ba62464bd16e456c 2023-03-30 03:56:45,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/851d6a2378c847c8a2da3adb09cd79cb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/851d6a2378c847c8a2da3adb09cd79cb 2023-03-30 03:56:45,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/44942974ed5f4346879a164e730bfc58 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/44942974ed5f4346879a164e730bfc58 2023-03-30 03:56:45,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/85e6adba8f3d4c1aa0d0a280e926408a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/85e6adba8f3d4c1aa0d0a280e926408a 2023-03-30 03:56:45,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/f958a8cc0c7844088c775a9d91af77cd to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/f958a8cc0c7844088c775a9d91af77cd 2023-03-30 03:56:45,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b49c3d9576544ee2986fb8835af69703 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b49c3d9576544ee2986fb8835af69703 2023-03-30 03:56:45,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3c78920eccf04dc4ba0a90d215d9d48e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/3c78920eccf04dc4ba0a90d215d9d48e 2023-03-30 03:56:45,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/6fb67444cd0f4925b0c3006a9519607b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/6fb67444cd0f4925b0c3006a9519607b 2023-03-30 03:56:45,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d83bd16d7f58468490ef07f9c90d02ae to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/d83bd16d7f58468490ef07f9c90d02ae 2023-03-30 03:56:45,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/181e7a7b223a4eee988a77bb21f152e7 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/181e7a7b223a4eee988a77bb21f152e7 2023-03-30 03:56:45,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c169bfe01098499ebb19da01700e7f8b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c169bfe01098499ebb19da01700e7f8b 2023-03-30 03:56:45,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fa2f3799c4db4fc5b2d5c6bdb654f525 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fa2f3799c4db4fc5b2d5c6bdb654f525 2023-03-30 03:56:45,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7f1cbee8d9c846b298dc4798ce3b3d81 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/7f1cbee8d9c846b298dc4798ce3b3d81 2023-03-30 03:56:45,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b978dfc96b984ba487adaa548a9d9149 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b978dfc96b984ba487adaa548a9d9149 2023-03-30 03:56:45,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/9d96da57589f46269dd0b0f669d5e247 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/9d96da57589f46269dd0b0f669d5e247 2023-03-30 03:56:45,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c0936686042c401490fc15dd91387595 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c0936686042c401490fc15dd91387595 2023-03-30 03:56:45,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2d21384507d04ad1b21b907fc65043d2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/2d21384507d04ad1b21b907fc65043d2 2023-03-30 03:56:45,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fbaee1f8780a43f5a65b2107eb6037fd to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/fbaee1f8780a43f5a65b2107eb6037fd 2023-03-30 03:56:45,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/5b7690ba4c6b471697fd72aaea520280 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/5b7690ba4c6b471697fd72aaea520280 2023-03-30 03:56:45,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/recovered.edits/606.seqid, newMaxSeqId=606, maxSeqId=1 2023-03-30 03:56:45,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f. 2023-03-30 03:56:45,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 3ecc6181b1966d72d93cc5e543ef979f: 2023-03-30 03:56:45,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:45,537 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3ecc6181b1966d72d93cc5e543ef979f, regionState=CLOSED 2023-03-30 03:56:45,538 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148605537"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148605537"}]},"ts":"1680148605537"} 2023-03-30 03:56:45,540 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-30 03:56:45,540 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 3ecc6181b1966d72d93cc5e543ef979f, server=jenkins-hbase20.apache.org,41157,1680148568851 in 1.2460 sec 2023-03-30 03:56:45,541 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-30 03:56:45,541 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3ecc6181b1966d72d93cc5e543ef979f, UNASSIGN in 1.2510 sec 2023-03-30 03:56:45,542 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148605542"}]},"ts":"1680148605542"} 2023-03-30 03:56:45,543 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 03:56:45,543 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 03:56:45,545 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.2900 sec 2023-03-30 03:56:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 03:56:46,373 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-30 03:56:46,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-30 03:56:46,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:46,389 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:46,391 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:46,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-30 03:56:46,395 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:46,399 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/recovered.edits] 2023-03-30 03:56:46,403 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/3b3438c0d8fe44dba146e1c75ae78338 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/3b3438c0d8fe44dba146e1c75ae78338 2023-03-30 03:56:46,404 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/6700a24c617b47cba0991527cf289c11 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/6700a24c617b47cba0991527cf289c11 2023-03-30 03:56:46,406 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/e87d58d67f174e31afb7891ae7b16922 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/A/e87d58d67f174e31afb7891ae7b16922 2023-03-30 03:56:46,408 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/079c808e26874461858ec054082c9866 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/079c808e26874461858ec054082c9866 2023-03-30 03:56:46,409 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3736687828b64ee9b1463596252fa4e6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/3736687828b64ee9b1463596252fa4e6 2023-03-30 03:56:46,410 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fd32937cbcaf4515bced9cd6768394fd to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/B/fd32937cbcaf4515bced9cd6768394fd 2023-03-30 03:56:46,412 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b95896f5e83d43dda0fe4d61881838a7 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/b95896f5e83d43dda0fe4d61881838a7 2023-03-30 03:56:46,413 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c08918ea03cd4bb19e3d0025f8cf0786 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c08918ea03cd4bb19e3d0025f8cf0786 2023-03-30 03:56:46,415 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c2b2021476334b2ca056aab5c210e837 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/C/c2b2021476334b2ca056aab5c210e837 2023-03-30 03:56:46,418 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/recovered.edits/606.seqid to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f/recovered.edits/606.seqid 2023-03-30 03:56:46,418 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/3ecc6181b1966d72d93cc5e543ef979f 2023-03-30 03:56:46,418 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:56:46,420 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:46,437 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 03:56:46,478 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 03:56:46,480 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:46,480 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 03:56:46,480 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680148606480"}]},"ts":"9223372036854775807"} 2023-03-30 03:56:46,482 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 03:56:46,482 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3ecc6181b1966d72d93cc5e543ef979f, NAME => 'TestAcidGuarantees,,1680148573183.3ecc6181b1966d72d93cc5e543ef979f.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 03:56:46,482 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 03:56:46,482 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680148606482"}]},"ts":"9223372036854775807"} 2023-03-30 03:56:46,484 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 03:56:46,486 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:46,487 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 102 msec 2023-03-30 03:56:46,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-30 03:56:46,494 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-30 03:56:46,512 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:41157-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xce2ac86-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xce2ac86-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0xce2ac86-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xce2ac86-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_191481841_15 at /127.0.0.1:55812 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=536 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=325 (was 171) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=10405 (was 10940) 2023-03-30 03:56:46,526 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=536, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=166, AvailableMemoryMB=10404 2023-03-30 03:56:46,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:56:46,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:56:46,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:46,531 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 03:56:46,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-30 03:56:46,532 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 03:56:46,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 03:56:46,533 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:46,534 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 empty. 2023-03-30 03:56:46,534 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:46,534 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:56:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 03:56:46,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 03:56:46,946 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:56:46,950 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 08f755044e3e0b03b5ae847b1c94e115, NAME => 'TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp 2023-03-30 03:56:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 03:56:47,365 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:47,366 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 08f755044e3e0b03b5ae847b1c94e115, disabling compactions & flushes 2023-03-30 03:56:47,366 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:47,366 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:47,366 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. after waiting 0 ms 2023-03-30 03:56:47,366 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:47,366 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:47,366 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:47,373 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 03:56:47,375 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148607375"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148607375"}]},"ts":"1680148607375"} 2023-03-30 03:56:47,376 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 03:56:47,377 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 03:56:47,378 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148607377"}]},"ts":"1680148607377"} 2023-03-30 03:56:47,379 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 03:56:47,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, ASSIGN}] 2023-03-30 03:56:47,383 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, ASSIGN 2023-03-30 03:56:47,384 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=false 2023-03-30 03:56:47,536 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=08f755044e3e0b03b5ae847b1c94e115, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:47,536 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148607535"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148607535"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148607535"}]},"ts":"1680148607535"} 2023-03-30 03:56:47,541 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:56:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 03:56:47,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:47,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 08f755044e3e0b03b5ae847b1c94e115, NAME => 'TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:56:47,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:47,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,706 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,709 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A 2023-03-30 03:56:47,709 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A 2023-03-30 03:56:47,709 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:47,710 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f755044e3e0b03b5ae847b1c94e115 columnFamilyName A 2023-03-30 03:56:47,711 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(310): Store=08f755044e3e0b03b5ae847b1c94e115/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:47,711 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,713 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B 2023-03-30 03:56:47,713 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B 2023-03-30 03:56:47,713 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:47,713 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f755044e3e0b03b5ae847b1c94e115 columnFamilyName B 2023-03-30 03:56:47,714 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(310): Store=08f755044e3e0b03b5ae847b1c94e115/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:47,714 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,716 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C 2023-03-30 03:56:47,716 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C 2023-03-30 03:56:47,717 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:47,717 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f755044e3e0b03b5ae847b1c94e115 columnFamilyName C 2023-03-30 03:56:47,718 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(310): Store=08f755044e3e0b03b5ae847b1c94e115/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:47,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:47,718 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:56:47,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:47,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:56:47,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 08f755044e3e0b03b5ae847b1c94e115; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60688977, jitterRate=-0.09566377103328705}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:56:47,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:47,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., pid=18, masterSystemTime=1680148607694 2023-03-30 03:56:47,726 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:47,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:47,726 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=08f755044e3e0b03b5ae847b1c94e115, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:47,726 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148607726"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148607726"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148607726"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148607726"}]},"ts":"1680148607726"} 2023-03-30 03:56:47,729 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-30 03:56:47,730 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 in 188 msec 2023-03-30 03:56:47,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-30 03:56:47,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, ASSIGN in 349 msec 2023-03-30 03:56:47,732 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 03:56:47,732 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148607732"}]},"ts":"1680148607732"} 2023-03-30 03:56:47,733 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 03:56:47,735 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 03:56:47,736 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2060 sec 2023-03-30 03:56:48,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 03:56:48,642 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-30 03:56:48,644 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x31388307 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:48,655 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f83e462, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:48,657 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:48,664 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47180, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:48,667 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 03:56:48,671 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40262, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 03:56:48,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:56:48,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:56:48,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-30 03:56:49,103 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 03:56:49,105 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:56:49,105 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 03:56:49,113 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-30 03:56:49,124 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, REOPEN/MOVE}] 2023-03-30 03:56:49,125 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, REOPEN/MOVE 2023-03-30 03:56:49,126 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=08f755044e3e0b03b5ae847b1c94e115, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,126 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148609126"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148609126"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148609126"}]},"ts":"1680148609126"} 2023-03-30 03:56:49,127 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:56:49,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 08f755044e3e0b03b5ae847b1c94e115, disabling compactions & flushes 2023-03-30 03:56:49,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. after waiting 0 ms 2023-03-30 03:56:49,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-30 03:56:49,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:49,285 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 08f755044e3e0b03b5ae847b1c94e115 to self. 2023-03-30 03:56:49,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,287 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=08f755044e3e0b03b5ae847b1c94e115, regionState=CLOSED 2023-03-30 03:56:49,287 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148609287"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148609287"}]},"ts":"1680148609287"} 2023-03-30 03:56:49,290 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-30 03:56:49,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 in 162 msec 2023-03-30 03:56:49,291 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=true 2023-03-30 03:56:49,442 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=08f755044e3e0b03b5ae847b1c94e115, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,442 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148609442"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148609442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148609442"}]},"ts":"1680148609442"} 2023-03-30 03:56:49,446 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:56:49,607 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,607 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 08f755044e3e0b03b5ae847b1c94e115, NAME => 'TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:56:49,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:56:49,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,612 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,614 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A 2023-03-30 03:56:49,614 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A 2023-03-30 03:56:49,614 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:49,617 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f755044e3e0b03b5ae847b1c94e115 columnFamilyName A 2023-03-30 03:56:49,619 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(310): Store=08f755044e3e0b03b5ae847b1c94e115/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:49,619 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,621 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B 2023-03-30 03:56:49,621 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B 2023-03-30 03:56:49,621 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:49,621 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f755044e3e0b03b5ae847b1c94e115 columnFamilyName B 2023-03-30 03:56:49,622 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(310): Store=08f755044e3e0b03b5ae847b1c94e115/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:49,622 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,623 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C 2023-03-30 03:56:49,623 DEBUG [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C 2023-03-30 03:56:49,623 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:56:49,624 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08f755044e3e0b03b5ae847b1c94e115 columnFamilyName C 2023-03-30 03:56:49,624 INFO [StoreOpener-08f755044e3e0b03b5ae847b1c94e115-1] regionserver.HStore(310): Store=08f755044e3e0b03b5ae847b1c94e115/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:56:49,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:56:49,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 08f755044e3e0b03b5ae847b1c94e115; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70015414, jitterRate=0.043310970067977905}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:56:49,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:49,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., pid=23, masterSystemTime=1680148609599 2023-03-30 03:56:49,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,635 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=08f755044e3e0b03b5ae847b1c94e115, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,635 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148609635"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148609635"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148609635"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148609635"}]},"ts":"1680148609635"} 2023-03-30 03:56:49,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-30 03:56:49,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 in 191 msec 2023-03-30 03:56:49,640 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-30 03:56:49,640 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, REOPEN/MOVE in 514 msec 2023-03-30 03:56:49,644 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-30 03:56:49,644 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 531 msec 2023-03-30 03:56:49,647 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 965 msec 2023-03-30 03:56:49,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-30 03:56:49,658 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x258ba157 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,663 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40eaa282, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,663 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x0df763e0 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,667 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19b40939, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,667 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x5649f410 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,671 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6017755a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,672 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x633c81ba to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,677 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38da9522, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,678 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x61ad5d1c to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,682 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@661b21c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,683 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x43a14bcf to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,688 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c0b7748, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,688 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x6577ce5a to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,693 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c411e08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,694 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x2d5bdceb to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,699 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7605c204, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,699 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x651f745a to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:56:49,710 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16505096, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:56:49,719 DEBUG [hconnection-0x5beddaaa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,719 DEBUG [hconnection-0x5dd28d9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,720 DEBUG [hconnection-0x2f1462d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:49,721 DEBUG [hconnection-0x7520d75b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,721 DEBUG [hconnection-0x245e78e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,729 DEBUG [hconnection-0x2944d7f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,729 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47186, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,729 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47210, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,729 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47184, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,730 DEBUG [hconnection-0x1431666b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,730 DEBUG [hconnection-0x158315a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,731 DEBUG [hconnection-0x7e55e776-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:56:49,731 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47246, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,733 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47216, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,733 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47230, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,733 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47258, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,736 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47290, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,736 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47260, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:56:49,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:49,747 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:49,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:49,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:49,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:49,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:49,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:49,750 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,750 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:49,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:49,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:49,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:49,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:49,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:49,760 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:49,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:49,760 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:49,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 03:56:49,761 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,761 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:49,761 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:49,761 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:49,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:49,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:49,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@452078b1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:49,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:49,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:49,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:49,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:49,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:49,765 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:49,765 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:49,765 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:49,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:49,767 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,767 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,767 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:49,767 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:49,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:49,769 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:49,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:49,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:49,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:49,769 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:49,771 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,771 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:49,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,771 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:49,772 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:49,774 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,782 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:49,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:49,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,782 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:49,782 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:49,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:49,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:49,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148669789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148669789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148669791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148669793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148669794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,882 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:49,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:49,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148669895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148669896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148669896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148669896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:49,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148669902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:49,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:49,904 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:49,904 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:49,904 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:49,905 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:49,905 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,905 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,907 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:49,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:49,907 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,908 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,908 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:49,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:49,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:49,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:49,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:49,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:49,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:49,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:49,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:49,915 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,915 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:49,915 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:49,915 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:49,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:49,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:49,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:49,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:49,917 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:49,918 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,918 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:49,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:49,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fb3eef6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:49,920 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:49,920 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,921 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:49,921 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:49,921 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:49,921 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,921 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:49,922 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,922 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:49,922 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:49,922 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:49,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:49,926 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:49,926 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:49,926 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:49,926 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:49,926 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:49,934 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,934 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,934 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:49,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:49,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,936 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:49,938 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:49,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:49,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:49,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:49,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:49,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:49,938 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:49,938 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:49,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:50,038 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:50,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:50,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:50,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:50,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:50,050 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:50,050 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:50,050 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:50,050 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:50,051 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,051 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:50,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:50,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:50,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:50,054 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:50,054 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:50,054 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:50,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:50,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:50,069 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:50,069 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:50,069 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:50,069 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:50,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:50,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:50,070 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,070 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:50,070 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:50,071 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:50,071 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,071 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:50,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:50,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea7eeba[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:50,073 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:50,073 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,074 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:50,074 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:50,074 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,074 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:50,074 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:50,075 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,075 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:50,075 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:50,076 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:50,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:50,078 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:50,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:50,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:50,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:50,078 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:50,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:50,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:50,082 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:50,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:50,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,084 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,084 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:50,085 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:50,085 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:50,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:50,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:50,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148670100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148670100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148670100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148670101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148670109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,185 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:50,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:50,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:50,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:50,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:50,195 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:50,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:50,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:50,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:50,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,196 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:50,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:50,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:50,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:50,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:50,203 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:50,203 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:50,203 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:50,203 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:50,209 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,209 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:50,209 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:50,209 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:50,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:50,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b1e9081[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:50,212 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:50,212 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,213 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,213 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:50,213 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:50,213 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:50,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:50,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:50,214 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:50,215 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,215 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:50,215 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:50,215 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:50,215 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:50,215 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:50,215 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:50,215 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:50,215 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303300e29bec612ed4e64b0b06b8b2d7748ad to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303300e29bec612ed4e64b0b06b8b2d7748ad 2023-03-30 03:56:50,216 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,216 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:50,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,217 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:50,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,217 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:50,219 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:50,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:50,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:50,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:50,221 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:50,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:50,222 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:50,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:50,222 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:50,222 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:50,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:50,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:50,244 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/72a4e115137d493e802dbf6f036acb3e 2023-03-30 03:56:50,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/7a8871ffabb243978dabdf05a62580db 2023-03-30 03:56:50,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/9c367223ee0f44a892210f88c3edd9db 2023-03-30 03:56:50,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/72a4e115137d493e802dbf6f036acb3e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/72a4e115137d493e802dbf6f036acb3e 2023-03-30 03:56:50,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/72a4e115137d493e802dbf6f036acb3e, entries=150, sequenceid=17, filesize=25.1 K 2023-03-30 03:56:50,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/7a8871ffabb243978dabdf05a62580db as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/7a8871ffabb243978dabdf05a62580db 2023-03-30 03:56:50,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/7a8871ffabb243978dabdf05a62580db, entries=150, sequenceid=17, filesize=11.6 K 2023-03-30 03:56:50,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/9c367223ee0f44a892210f88c3edd9db as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c367223ee0f44a892210f88c3edd9db 2023-03-30 03:56:50,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c367223ee0f44a892210f88c3edd9db, entries=150, sequenceid=17, filesize=11.6 K 2023-03-30 03:56:50,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 08f755044e3e0b03b5ae847b1c94e115 in 561ms, sequenceid=17, compaction requested=false 2023-03-30 03:56:50,321 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-30 03:56:50,322 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:50,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:50,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:50,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:50,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:50,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:50,329 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:50,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:50,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:50,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:50,330 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,330 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:50,331 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:50,331 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,331 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:50,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:50,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:50,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:50,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:50,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:50,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:50,337 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:50,337 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:50,337 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:50,337 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:50,337 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 03:56:50,338 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:50,338 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:50,338 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:50,338 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:50,338 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:50,338 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:50,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:50,406 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:50,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148670412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148670412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148670413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148670415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148670415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148670519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148670519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148670519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148670522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148670522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148670724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148670724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148670725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148670726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:50,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148670726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:50,764 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330e67e1e987a2d460581987c006ef8f056 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330e67e1e987a2d460581987c006ef8f056 2023-03-30 03:56:51,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148671027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148671027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148671028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148671028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148671029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,089 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 03:56:51,169 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/54b242ed2ff64545b0d438f08d897e61 2023-03-30 03:56:51,196 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/b400ec351baf4ad887909cb5565b4d51 2023-03-30 03:56:51,212 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/04ecc2b5286a48779eff5395e353271c 2023-03-30 03:56:51,219 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/54b242ed2ff64545b0d438f08d897e61 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/54b242ed2ff64545b0d438f08d897e61 2023-03-30 03:56:51,226 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/54b242ed2ff64545b0d438f08d897e61, entries=150, sequenceid=36, filesize=25.1 K 2023-03-30 03:56:51,227 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/b400ec351baf4ad887909cb5565b4d51 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b400ec351baf4ad887909cb5565b4d51 2023-03-30 03:56:51,233 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b400ec351baf4ad887909cb5565b4d51, entries=150, sequenceid=36, filesize=11.6 K 2023-03-30 03:56:51,235 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/04ecc2b5286a48779eff5395e353271c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/04ecc2b5286a48779eff5395e353271c 2023-03-30 03:56:51,239 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/04ecc2b5286a48779eff5395e353271c, entries=150, sequenceid=36, filesize=11.6 K 2023-03-30 03:56:51,240 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 08f755044e3e0b03b5ae847b1c94e115 in 903ms, sequenceid=36, compaction requested=false 2023-03-30 03:56:51,240 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:51,240 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,240 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:51,240 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:51,240 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,243 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:51,243 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:51,244 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,244 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:51,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dbd82a3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:51,246 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:51,246 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,247 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,247 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:51,247 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:51,247 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:51,248 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,248 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:51,248 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:51,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:51,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:51,253 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:51,253 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:51,253 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:51,253 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:51,253 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:51,255 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,255 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,256 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,259 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,267 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:51,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:51,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,268 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:51,268 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:51,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,368 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:51,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:51,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:51,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:51,372 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:51,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:51,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:51,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:51,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,373 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:51,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:51,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:51,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:51,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:51,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:51,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:51,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:51,374 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,374 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:51,374 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:56:51,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:51,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:51,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:51,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:51,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:51,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:51,401 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d19d91ce11e1499999da90fe9ae4f365 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d19d91ce11e1499999da90fe9ae4f365 2023-03-30 03:56:51,408 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/bd836302b0474427901e62e3885e4d90 2023-03-30 03:56:51,436 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/67021198b3a142729b3d724ae61842e2 2023-03-30 03:56:51,467 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/885b3a0321a544d495ebc2df7725698a 2023-03-30 03:56:51,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/bd836302b0474427901e62e3885e4d90 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bd836302b0474427901e62e3885e4d90 2023-03-30 03:56:51,477 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bd836302b0474427901e62e3885e4d90, entries=150, sequenceid=49, filesize=25.1 K 2023-03-30 03:56:51,482 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/67021198b3a142729b3d724ae61842e2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/67021198b3a142729b3d724ae61842e2 2023-03-30 03:56:51,486 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/67021198b3a142729b3d724ae61842e2, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 03:56:51,487 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/885b3a0321a544d495ebc2df7725698a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/885b3a0321a544d495ebc2df7725698a 2023-03-30 03:56:51,492 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/885b3a0321a544d495ebc2df7725698a, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 03:56:51,493 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 08f755044e3e0b03b5ae847b1c94e115 in 119ms, sequenceid=49, compaction requested=true 2023-03-30 03:56:51,493 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:51,493 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,493 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:51,495 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,495 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,495 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,495 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:51,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6917bc[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:51,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:51,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:51,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:51,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:51,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:51,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:51,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:51,502 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:51,502 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:51,502 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:51,502 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:51,502 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:51,504 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,504 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,504 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,505 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:51,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,508 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,508 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:51,508 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:51,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:56:51,600 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:51,600 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:51,600 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:51,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:51,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:51,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:51,608 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:51,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:51,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:51,613 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:51,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:51,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:51,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:51,614 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,614 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,614 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:51,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:51,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:51,625 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:51,625 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:51,625 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,625 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:51,625 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:51,625 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,625 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:51,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:51,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:51,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:51,627 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,628 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,628 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:51,629 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:51,630 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:51,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b6df5f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:51,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:51,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680148611601, max timestamp=1680148611630Num uniques 859; 2023-03-30 03:56:51,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680148611601, max timestamp=1680148611630Num uniques 859; 2023-03-30 03:56:51,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680148611601, max timestamp=1680148611630Num uniques 859; 2023-03-30 03:56:51,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 03:56:51,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 03:56:51,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 03:56:51,633 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:51,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:51,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:51,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:51,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,634 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:51,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:51,637 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:51,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:51,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:51,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:51,638 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:51,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148671636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148671637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148671639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,645 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:51,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:51,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,645 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:51,645 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:51,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:51,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148671641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148671641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148671742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148671744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148671745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,745 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:51,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:51,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148671748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:51,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:51,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,754 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:51,754 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:51,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148671749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,754 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:51,754 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:51,755 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,755 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,755 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,755 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:51,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:51,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:51,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:51,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:51,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:51,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:51,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:51,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:51,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:51,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:51,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:51,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:51,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:51,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:51,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7747f642[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:51,777 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:51,777 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,777 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:51,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:51,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:51,778 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,778 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:51,778 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:51,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,778 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:51,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:51,781 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:51,781 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:51,781 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:51,781 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:51,781 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:51,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,782 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,786 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,786 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:51,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,786 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:51,886 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:51,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:51,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:51,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:51,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:51,890 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:51,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:51,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:51,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:51,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,891 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,892 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:51,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:51,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:51,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:51,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:51,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:51,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:51,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:51,892 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,893 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:51,893 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:51,893 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:51,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,894 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,894 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:51,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cf2c861[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:51,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:51,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,895 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:51,896 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,896 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:51,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:51,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:51,896 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:51,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,897 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:51,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:51,898 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:51,899 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:51,899 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:51,899 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:51,899 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:51,899 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,899 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:51,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,900 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:51,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,900 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,902 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:51,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:51,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:51,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:51,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:51,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:51,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:51,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:51,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:51,903 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:51,904 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:51,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:51,904 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:51,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148671947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148671947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148671951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148671955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:51,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:51,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148671956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,004 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,007 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,008 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,009 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,009 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,009 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,010 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,010 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,010 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,010 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,010 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,010 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,011 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,012 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1069b7a5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,014 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,016 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,016 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,017 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,018 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,018 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,018 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,020 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,021 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d4e332f3f86140b98fea8f8c2f2d6af4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d4e332f3f86140b98fea8f8c2f2d6af4 2023-03-30 03:56:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,023 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,024 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,024 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:52,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,124 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,129 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,131 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,131 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,132 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,132 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,141 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,141 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,141 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,141 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c1ded0[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,144 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,144 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,145 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,145 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,145 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,146 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,146 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,146 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,146 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,149 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,149 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,151 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,152 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,153 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,155 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,155 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,155 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,155 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,155 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,155 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,155 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:52,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,156 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,156 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148672251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148672253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,256 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148672254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,259 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,259 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,259 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,259 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148672259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148672263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,265 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,266 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,266 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,266 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,267 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,267 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,267 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,267 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,267 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,267 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,267 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,267 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,267 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,267 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,267 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,268 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,269 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,269 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7cd3ec[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,271 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,271 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,271 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,271 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,271 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,271 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,276 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,276 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,283 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,283 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,283 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,283 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,287 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,291 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,291 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:52,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,391 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,394 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,396 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,396 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,396 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,396 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,398 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f7b5c2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,399 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,399 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,400 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,400 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,400 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,400 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,401 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,401 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,401 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,403 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,403 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,403 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,403 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,403 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,404 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,405 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,407 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,407 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,427 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/e83a79c82ce34239a0e5535c0d97299a 2023-03-30 03:56:52,507 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,511 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,512 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,512 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,512 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,512 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,513 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,513 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,513 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,513 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,513 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,513 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,513 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,514 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,514 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,515 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b7eee26[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,517 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,517 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,518 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,519 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,522 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,522 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,527 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,527 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,627 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,631 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,632 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,634 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,634 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,634 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,634 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,635 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21b39cc7[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,640 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,640 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,641 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,641 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,641 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:52,644 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,645 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,645 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,745 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,748 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,750 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,750 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,750 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,750 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b0ac8ae[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,753 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,753 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,753 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,754 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148672754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148672756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,756 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,756 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,759 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,760 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148672759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,762 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,762 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,763 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,763 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,763 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:52,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148672765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:52,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148672765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/6abbe1929db8451f9b7e1b2cd9ebfc14 2023-03-30 03:56:52,862 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,866 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,866 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,866 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,866 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,866 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,866 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,867 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,867 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,867 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,867 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,868 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,869 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,869 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,869 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,870 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40683ba8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,872 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,873 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,873 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,873 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,877 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,877 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,877 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,877 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,877 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,877 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,877 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,877 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,878 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,881 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,881 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,881 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,881 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,981 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:52,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:52,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:52,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:52,986 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:52,986 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,986 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:52,986 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:52,986 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,986 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:52,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:52,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:52,988 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:52,988 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:52,988 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:52,988 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:52,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:52,989 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,989 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:52,989 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:52,989 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:52,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:52,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:52,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:52,989 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:52,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:52,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bfa8f8e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:52,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:52,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:52,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:52,991 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:52,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:52,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:52,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:52,992 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:52,992 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,992 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,992 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:52,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:52,995 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:52,995 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:52,995 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:52,995 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:52,995 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:52,996 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,996 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,996 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:52,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,997 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:52,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:52,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:52,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:52,999 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:52,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:52,999 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:52,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:52,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,099 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:53,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:53,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:53,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:53,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:53,102 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:53,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:53,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:53,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:53,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,103 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,103 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,103 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:53,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:53,103 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,103 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:53,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:53,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:53,104 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:53,104 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:53,105 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:53,106 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:53,106 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:53,106 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:53,106 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:53,106 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:53,106 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:53,106 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:53,106 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:53,106 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,106 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:53,106 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:53,107 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,107 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:53,107 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,107 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:53,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:53,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c4be25d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:53,109 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:53,109 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,109 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:53,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:53,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:53,109 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,110 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:53,110 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:53,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:53,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:53,113 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:53,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:53,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:53,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:53,113 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:53,114 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,114 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,114 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:53,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:53,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:53,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:53,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:53,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:53,117 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,117 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,117 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:53,117 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,117 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:53,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:53,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:53,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:53,217 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:53,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:53,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:53,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:53,220 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:53,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:53,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:53,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:53,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:53,221 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:53,221 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,221 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:53,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:53,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:53,222 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:53,222 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:53,222 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:53,222 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:53,222 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:53,222 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:53,222 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:53,222 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:53,222 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:53,223 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:53,223 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:53,223 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,223 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:53,223 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:53,223 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,224 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:53,225 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,225 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a31eb7[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:53,226 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:53,226 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,227 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:53,227 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,227 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:53,228 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,228 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:53,228 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:53,228 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:53,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:53,230 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:53,230 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:53,230 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:53,230 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:53,230 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:53,231 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,231 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:53,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,232 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:53,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:53,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:53,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,235 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:53,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:53,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:53,235 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:53,235 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:53,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:53,236 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:53,236 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:53,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:53,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/d376edf12b614a7baab887eac477740e 2023-03-30 03:56:53,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/e83a79c82ce34239a0e5535c0d97299a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e83a79c82ce34239a0e5535c0d97299a 2023-03-30 03:56:53,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e83a79c82ce34239a0e5535c0d97299a, entries=400, sequenceid=60, filesize=59.0 K 2023-03-30 03:56:53,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/6abbe1929db8451f9b7e1b2cd9ebfc14 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6abbe1929db8451f9b7e1b2cd9ebfc14 2023-03-30 03:56:53,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6abbe1929db8451f9b7e1b2cd9ebfc14, entries=150, sequenceid=60, filesize=11.6 K 2023-03-30 03:56:53,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/d376edf12b614a7baab887eac477740e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d376edf12b614a7baab887eac477740e 2023-03-30 03:56:53,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d376edf12b614a7baab887eac477740e, entries=150, sequenceid=60, filesize=11.6 K 2023-03-30 03:56:53,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=154.31 KB/158010 for 08f755044e3e0b03b5ae847b1c94e115 in 1678ms, sequenceid=60, compaction requested=true 2023-03-30 03:56:53,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:53,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:53,277 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:53,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:53,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-30 03:56:53,278 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:53,282 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 137692 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:53,283 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:53,283 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/A is initiating minor compaction (all files) 2023-03-30 03:56:53,283 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/B is initiating minor compaction (all files) 2023-03-30 03:56:53,283 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/A in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:53,283 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/B in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:53,283 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/7a8871ffabb243978dabdf05a62580db, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b400ec351baf4ad887909cb5565b4d51, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/67021198b3a142729b3d724ae61842e2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6abbe1929db8451f9b7e1b2cd9ebfc14] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=46.4 K 2023-03-30 03:56:53,283 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/72a4e115137d493e802dbf6f036acb3e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/54b242ed2ff64545b0d438f08d897e61, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bd836302b0474427901e62e3885e4d90, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e83a79c82ce34239a0e5535c0d97299a] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=134.5 K 2023-03-30 03:56:53,283 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 7a8871ffabb243978dabdf05a62580db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680148609757 2023-03-30 03:56:53,283 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 72a4e115137d493e802dbf6f036acb3e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680148609757 2023-03-30 03:56:53,284 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b400ec351baf4ad887909cb5565b4d51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680148609789 2023-03-30 03:56:53,284 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 54b242ed2ff64545b0d438f08d897e61, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680148609789 2023-03-30 03:56:53,284 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 67021198b3a142729b3d724ae61842e2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148610411 2023-03-30 03:56:53,284 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting bd836302b0474427901e62e3885e4d90, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148610411 2023-03-30 03:56:53,284 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting e83a79c82ce34239a0e5535c0d97299a, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680148611588 2023-03-30 03:56:53,284 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6abbe1929db8451f9b7e1b2cd9ebfc14, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680148611592 2023-03-30 03:56:53,292 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#B#compaction#141 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:53,321 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#A#compaction#142 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:53,330 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/4464394dbe3c4dd8b2d92120aac76220 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4464394dbe3c4dd8b2d92120aac76220 2023-03-30 03:56:53,335 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:53,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:53,338 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/B of 08f755044e3e0b03b5ae847b1c94e115 into 4464394dbe3c4dd8b2d92120aac76220(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:53,338 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:53,338 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/B, priority=12, startTime=1680148613277; duration=0sec 2023-03-30 03:56:53,339 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:53,339 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:53,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:53,341 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:53,342 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/C is initiating minor compaction (all files) 2023-03-30 03:56:53,342 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/C in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:53,342 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c367223ee0f44a892210f88c3edd9db, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/04ecc2b5286a48779eff5395e353271c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/885b3a0321a544d495ebc2df7725698a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d376edf12b614a7baab887eac477740e] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=46.4 K 2023-03-30 03:56:53,343 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 9c367223ee0f44a892210f88c3edd9db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680148609757 2023-03-30 03:56:53,343 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 04ecc2b5286a48779eff5395e353271c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680148609789 2023-03-30 03:56:53,344 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 885b3a0321a544d495ebc2df7725698a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148610411 2023-03-30 03:56:53,344 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d376edf12b614a7baab887eac477740e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680148611592 2023-03-30 03:56:53,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:53,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:53,354 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:53,354 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:53,354 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:53,354 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:53,354 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,354 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,355 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:53,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:53,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:53,356 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#C#compaction#143 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:53,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:53,357 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,357 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:53,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:53,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:53,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:53,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:53,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:53,363 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:53,363 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:53,369 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:53,370 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:53,370 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-03-30 03:56:53,370 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:53,370 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:53,370 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:53,370 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:53,370 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:53,370 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:53,387 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330fb626ca6791f4c6595564c789cf01d34 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330fb626ca6791f4c6595564c789cf01d34 2023-03-30 03:56:53,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:53,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:53,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:53,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148673767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:53,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148673771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:53,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148673771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148673777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:53,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148673777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,783 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/df1e5b7ac6c8449c89660a2a655da07b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df1e5b7ac6c8449c89660a2a655da07b 2023-03-30 03:56:53,788 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/C of 08f755044e3e0b03b5ae847b1c94e115 into df1e5b7ac6c8449c89660a2a655da07b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:53,788 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:53,788 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/C, priority=12, startTime=1680148613277; duration=0sec 2023-03-30 03:56:53,788 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:53,792 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=87, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/01c93affa38c4cb1b950da3bd7990943 2023-03-30 03:56:53,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:53,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148673872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:53,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:53,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148673872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:54,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148674073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:54,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148674079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,182 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/6d440c9507cb49fb908d4f3e24b2ed30 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6d440c9507cb49fb908d4f3e24b2ed30 2023-03-30 03:56:54,187 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/A of 08f755044e3e0b03b5ae847b1c94e115 into 6d440c9507cb49fb908d4f3e24b2ed30(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:54,187 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:54,187 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/A, priority=12, startTime=1680148613277; duration=0sec 2023-03-30 03:56:54,187 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:54,204 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/0d2d0d097a5745bd98a9b307100eeadb 2023-03-30 03:56:54,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:54,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148674378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:54,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148674382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,615 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/541317265be04f31a766dbbe2c42238a 2023-03-30 03:56:54,619 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/01c93affa38c4cb1b950da3bd7990943 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/01c93affa38c4cb1b950da3bd7990943 2023-03-30 03:56:54,622 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/01c93affa38c4cb1b950da3bd7990943, entries=150, sequenceid=87, filesize=25.1 K 2023-03-30 03:56:54,623 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/0d2d0d097a5745bd98a9b307100eeadb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/0d2d0d097a5745bd98a9b307100eeadb 2023-03-30 03:56:54,632 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/0d2d0d097a5745bd98a9b307100eeadb, entries=150, sequenceid=87, filesize=11.6 K 2023-03-30 03:56:54,633 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/541317265be04f31a766dbbe2c42238a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/541317265be04f31a766dbbe2c42238a 2023-03-30 03:56:54,637 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/541317265be04f31a766dbbe2c42238a, entries=150, sequenceid=87, filesize=11.6 K 2023-03-30 03:56:54,637 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=46.96 KB/48090 for 08f755044e3e0b03b5ae847b1c94e115 in 1267ms, sequenceid=87, compaction requested=false 2023-03-30 03:56:54,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:54,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:54,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:54,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:54,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:54,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:54,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:54,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:54,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:54,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:54,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cc1f395[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:54,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:54,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:54,641 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:54,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:54,641 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:54,641 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:54,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:54,641 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:54,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:54,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:54,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:54,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:54,645 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:54,645 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:54,645 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:54,645 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:54,645 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:54,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:54,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:54,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:54,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,647 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:54,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:54,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:54,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,664 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:54,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:54,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:54,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:54,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:54,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:54,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:54,665 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:54,665 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:54,665 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:54,665 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:54,665 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:54,765 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:54,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:54,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:54,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:54,769 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:54,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:54,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:54,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:54,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:54,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:54,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:54,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,771 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:54,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:54,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:54,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:54,772 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:54,772 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:54,772 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:54,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:54,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:54,774 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:54,774 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:54,774 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:56:54,774 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:54,775 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:54,775 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:54,775 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:54,775 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:54,775 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:54,792 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303303143310f7bad44e1afe9b3e8cb90102c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303143310f7bad44e1afe9b3e8cb90102c 2023-03-30 03:56:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:54,897 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:54,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148614908, max timestamp=1680148614929Num uniques 859; 2023-03-30 03:56:54,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:56:54,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148614908, max timestamp=1680148614929Num uniques 859; 2023-03-30 03:56:54,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:56:54,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148614908, max timestamp=1680148614929Num uniques 859; 2023-03-30 03:56:54,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:56:54,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148674953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:54,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148674956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148675057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148675059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,210 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/51808ce9c1f7472e9e133ef9954a5b30 2023-03-30 03:56:55,219 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/24670a1e6f4241aeb14a86e0c2c1bd27 2023-03-30 03:56:55,236 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/134166b58b72430fb78566ab3682c204 2023-03-30 03:56:55,240 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/51808ce9c1f7472e9e133ef9954a5b30 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/51808ce9c1f7472e9e133ef9954a5b30 2023-03-30 03:56:55,243 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/51808ce9c1f7472e9e133ef9954a5b30, entries=100, sequenceid=99, filesize=18.4 K 2023-03-30 03:56:55,245 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/24670a1e6f4241aeb14a86e0c2c1bd27 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/24670a1e6f4241aeb14a86e0c2c1bd27 2023-03-30 03:56:55,251 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/24670a1e6f4241aeb14a86e0c2c1bd27, entries=100, sequenceid=99, filesize=9.3 K 2023-03-30 03:56:55,252 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/134166b58b72430fb78566ab3682c204 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/134166b58b72430fb78566ab3682c204 2023-03-30 03:56:55,256 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/134166b58b72430fb78566ab3682c204, entries=100, sequenceid=99, filesize=9.3 K 2023-03-30 03:56:55,257 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 08f755044e3e0b03b5ae847b1c94e115 in 483ms, sequenceid=99, compaction requested=true 2023-03-30 03:56:55,257 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:55,257 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:55,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:55,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:55,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:55,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,259 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,260 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,260 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:55,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d7522de[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:55,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:55,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,262 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:55,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:55,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:55,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:55,263 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:55,263 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,263 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,263 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-30 03:56:55,263 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:55,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:55,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:55,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:55,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:55,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:55,265 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:55,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:55,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:55,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:55,265 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:55,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,267 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,269 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,274 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:55,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:55,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:55,274 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:55,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,274 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,274 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:55,275 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,275 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,275 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148675282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148675282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,375 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:55,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:55,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:55,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:55,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:55,377 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:55,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:55,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:55,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,379 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:55,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:55,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:55,380 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,380 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:55,380 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:55,380 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:55,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:55,381 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:55,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e1f6e27[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:55,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:55,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:55,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:55,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:55,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:55,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,384 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:55,384 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,384 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:55,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148675384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148675384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:55,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:55,391 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:55,391 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:55,391 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:55,391 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:55,391 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:55,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,404 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:55,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:55,404 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:55,404 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:55,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:55,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,406 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,406 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,505 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:55,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:55,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:55,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:55,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:55,510 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:55,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:55,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:55,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:55,513 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:55,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:55,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:55,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:55,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:55,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:55,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:55,515 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,515 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:55,515 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:55,515 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:55,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:55,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:55,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:55,516 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:55,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78087f5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:55,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:55,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,518 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:55,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:55,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:55,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:55,519 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:55,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:55,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:55,522 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:55,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:55,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:55,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:55,522 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:55,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:55,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:55,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,528 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:55,528 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:55,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:55,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148675586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148675587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,628 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:55,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:55,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:55,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:55,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:55,631 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:55,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:55,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:55,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,632 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:55,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:55,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:55,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:55,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:55,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:55,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:55,634 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,634 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:55,634 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:55,634 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:55,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:55,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:55,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:55,635 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:55,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742e81cb[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:55,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:55,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:55,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:55,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:55,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:55,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:55,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:55,640 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:55,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:55,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:55,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:55,640 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:55,641 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,641 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,641 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:55,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:55,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:55,644 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,644 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:55,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,644 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:55,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,684 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033038d7b12a18894dac92d38b6777cd3c9b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033038d7b12a18894dac92d38b6777cd3c9b 2023-03-30 03:56:55,744 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:55,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:55,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:55,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:55,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:55,747 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:55,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:55,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:55,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:55,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:55,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:55,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:55,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:55,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:55,753 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,753 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:55,753 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:55,753 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:55,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,754 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,755 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,755 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:55,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20192e42[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:55,757 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:55,757 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,757 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,758 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,758 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:55,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:55,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:55,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:55,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:55,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:55,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:55,761 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:55,761 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:55,761 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:55,761 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:55,761 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:55,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,764 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,764 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,765 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,775 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:55,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,775 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,775 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:55,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148675780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148675783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,785 DEBUG [Thread-1005] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:55,786 DEBUG [Thread-1009] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:55,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148675788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,790 DEBUG [Thread-1007] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:55,876 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:55,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:55,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:55,879 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:55,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:55,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:55,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,881 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,881 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:55,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:55,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:55,882 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,882 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:55,882 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:55,882 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:55,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:55,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,884 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:55,885 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15420230[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:55,885 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,885 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,886 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,886 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:55,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:55,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:55,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:55,886 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:55,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:55,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:55,888 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:55,888 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:55,888 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:55,888 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:55,888 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:55,889 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,889 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,889 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,889 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148675889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,892 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:55,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:55,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:55,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:55,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,892 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:55,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:55,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148675890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:55,892 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:55,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:55,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,992 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:55,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:55,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:55,996 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:55,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:55,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:55,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,997 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,997 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:55,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:55,997 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,997 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:55,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:55,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:55,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:55,998 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,998 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:55,998 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:55,998 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:55,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,999 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:56,001 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:56,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16fe8042[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:56,001 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,001 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:56,001 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,001 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:56,001 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:56,001 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:56,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:56,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:56,006 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:56,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:56,008 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:56,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:56,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:56,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:56,008 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:56,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,009 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:56,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:56,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:56,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,012 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:56,012 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,012 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:56,012 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,012 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,012 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,012 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,089 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/768a3e4c1cc94d52b92b3a676596833e 2023-03-30 03:56:56,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/77eb8a5ff61042b3a27687f186faf56d 2023-03-30 03:56:56,112 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:56,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:56,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:56,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:56,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:56,115 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:56,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:56,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:56,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:56,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:56,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:56,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:56,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:56,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:56,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:56,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:56,117 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,117 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:56,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:56,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:56,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:56,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:56,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:56,118 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:56,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:56,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@165e6b8f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:56,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,121 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:56,121 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:56,121 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:56,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:56,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:56,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:56,123 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:56,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:56,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:56,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:56,124 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:56,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,125 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,126 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,127 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,127 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:56,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:56,128 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,128 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:56,128 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:56,128 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,128 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,128 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,128 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,128 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,128 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,128 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,228 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:56,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:56,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:56,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:56,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:56,231 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:56,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:56,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:56,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,232 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:56,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:56,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:56,233 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,233 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:56,233 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:56,233 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:56,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:56,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,235 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,235 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,235 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:56,236 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:56,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5c3c38[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:56,236 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,236 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:56,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:56,237 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,237 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:56,237 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,237 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:56,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:56,239 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:56,239 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:56,239 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:56,239 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:56,239 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:56,240 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,240 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,240 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,240 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,241 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,243 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:56,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,243 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:56,243 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:56,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,343 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:56,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:56,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:56,346 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:56,346 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,346 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:56,346 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:56,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:56,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:56,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:56,349 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:56,349 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,349 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:56,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:56,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:56,350 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,350 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:56,350 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:56,350 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:56,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:56,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:56,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:56,351 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,351 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,351 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,352 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:56,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8ae261[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:56,352 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:56,353 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,353 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,353 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:56,353 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:56,353 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:56,353 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:56,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,354 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:56,354 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,354 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:56,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:56,356 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:56,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:56,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:56,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:56,356 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:56,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,358 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,358 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,359 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:56,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:56,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:56,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,360 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:56,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,360 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:56,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:56,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148676395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:56,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148676397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,460 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:56,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:56,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:56,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:56,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:56,463 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:56,463 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,463 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:56,463 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:56,463 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,463 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,465 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,465 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,465 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:56,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:56,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:56,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:56,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:56,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:56,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:56,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:56,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:56,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:56,467 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:56,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:56,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:56,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:56,468 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:56,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@181491e2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:56,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:56,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,470 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:56,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:56,470 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:56,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:56,472 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:56,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:56,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:56,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:56,472 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:56,473 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,473 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,473 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,473 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:56,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,475 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:56,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:56,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:56,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:56,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:56,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:56,476 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:56,476 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:56,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:56,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/e91773e578a44a0e811ea0f54b57f0b6 2023-03-30 03:56:56,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/768a3e4c1cc94d52b92b3a676596833e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/768a3e4c1cc94d52b92b3a676596833e 2023-03-30 03:56:56,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/768a3e4c1cc94d52b92b3a676596833e, entries=200, sequenceid=136, filesize=32.1 K 2023-03-30 03:56:56,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/77eb8a5ff61042b3a27687f186faf56d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/77eb8a5ff61042b3a27687f186faf56d 2023-03-30 03:56:56,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/77eb8a5ff61042b3a27687f186faf56d, entries=150, sequenceid=136, filesize=11.7 K 2023-03-30 03:56:56,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/e91773e578a44a0e811ea0f54b57f0b6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/e91773e578a44a0e811ea0f54b57f0b6 2023-03-30 03:56:56,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/e91773e578a44a0e811ea0f54b57f0b6, entries=150, sequenceid=136, filesize=11.7 K 2023-03-30 03:56:56,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 08f755044e3e0b03b5ae847b1c94e115 in 1263ms, sequenceid=136, compaction requested=true 2023-03-30 03:56:56,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:56,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:56,527 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:56,527 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:56,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:56,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:56,528 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103264 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:56,528 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45438 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:56,528 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/A is initiating minor compaction (all files) 2023-03-30 03:56:56,528 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/B is initiating minor compaction (all files) 2023-03-30 03:56:56,528 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/A in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,528 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/B in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,528 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6d440c9507cb49fb908d4f3e24b2ed30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/01c93affa38c4cb1b950da3bd7990943, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/51808ce9c1f7472e9e133ef9954a5b30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/768a3e4c1cc94d52b92b3a676596833e] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=100.8 K 2023-03-30 03:56:56,528 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4464394dbe3c4dd8b2d92120aac76220, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/0d2d0d097a5745bd98a9b307100eeadb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/24670a1e6f4241aeb14a86e0c2c1bd27, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/77eb8a5ff61042b3a27687f186faf56d] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=44.4 K 2023-03-30 03:56:56,529 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4464394dbe3c4dd8b2d92120aac76220, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680148611592 2023-03-30 03:56:56,529 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 6d440c9507cb49fb908d4f3e24b2ed30, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680148611592 2023-03-30 03:56:56,530 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 0d2d0d097a5745bd98a9b307100eeadb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680148611629 2023-03-30 03:56:56,530 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 01c93affa38c4cb1b950da3bd7990943, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680148611629 2023-03-30 03:56:56,530 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 51808ce9c1f7472e9e133ef9954a5b30, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680148613763 2023-03-30 03:56:56,534 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 24670a1e6f4241aeb14a86e0c2c1bd27, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680148613763 2023-03-30 03:56:56,534 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 77eb8a5ff61042b3a27687f186faf56d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1680148614936 2023-03-30 03:56:56,535 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 768a3e4c1cc94d52b92b3a676596833e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1680148614936 2023-03-30 03:56:56,545 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#B#compaction#153 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:56,564 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#A#compaction#154 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:56,576 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:56,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:56,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:56,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:56,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:56,583 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:56,583 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,583 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:56,583 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:56,584 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,584 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,584 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:56,584 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:56,585 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:56,585 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:56,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:56,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:56,585 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:56,585 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:56,586 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:56,586 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,586 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:56,586 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-30 03:56:56,587 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:56,587 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:56,587 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:56,587 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:56,587 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:56,587 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:56,966 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/4d09419acffc4caa9878056c285b669b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4d09419acffc4caa9878056c285b669b 2023-03-30 03:56:56,972 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/B of 08f755044e3e0b03b5ae847b1c94e115 into 4d09419acffc4caa9878056c285b669b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:56,972 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:56,972 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/B, priority=12, startTime=1680148616527; duration=0sec 2023-03-30 03:56:56,972 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:56,972 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:56:56,975 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45438 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:56:56,975 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/C is initiating minor compaction (all files) 2023-03-30 03:56:56,975 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/C in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:56,975 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df1e5b7ac6c8449c89660a2a655da07b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/541317265be04f31a766dbbe2c42238a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/134166b58b72430fb78566ab3682c204, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/e91773e578a44a0e811ea0f54b57f0b6] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=44.4 K 2023-03-30 03:56:56,975 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting df1e5b7ac6c8449c89660a2a655da07b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680148611592 2023-03-30 03:56:56,976 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 541317265be04f31a766dbbe2c42238a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680148611629 2023-03-30 03:56:56,976 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 134166b58b72430fb78566ab3682c204, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680148613763 2023-03-30 03:56:56,976 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e91773e578a44a0e811ea0f54b57f0b6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1680148614936 2023-03-30 03:56:56,985 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#C#compaction#156 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:56,998 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033051b6ee94af7443d794bcaeaecf625998 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033051b6ee94af7443d794bcaeaecf625998 2023-03-30 03:56:57,003 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=145, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/9bc713a811c54ce1aa3134b762571533 2023-03-30 03:56:57,395 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/b3429102086b4beeaa45ee72a3b985eb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b3429102086b4beeaa45ee72a3b985eb 2023-03-30 03:56:57,400 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/C of 08f755044e3e0b03b5ae847b1c94e115 into b3429102086b4beeaa45ee72a3b985eb(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:57,400 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:57,400 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/C, priority=12, startTime=1680148616527; duration=0sec 2023-03-30 03:56:57,400 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:57,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:57,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:57,414 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/4f2331768da844aca7001af4d006d120 2023-03-30 03:56:57,444 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/9c9964df93b74471a0de8200115b8514 2023-03-30 03:56:57,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148617432, max timestamp=1680148617443Num uniques 859; 2023-03-30 03:56:57,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148617432, max timestamp=1680148617443Num uniques 859; 2023-03-30 03:56:57,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:57,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148617432, max timestamp=1680148617443Num uniques 859; 2023-03-30 03:56:57,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:57,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:56:57,449 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/9bc713a811c54ce1aa3134b762571533 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/9bc713a811c54ce1aa3134b762571533 2023-03-30 03:56:57,453 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/9bc713a811c54ce1aa3134b762571533, entries=150, sequenceid=145, filesize=25.3 K 2023-03-30 03:56:57,456 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/4f2331768da844aca7001af4d006d120 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4f2331768da844aca7001af4d006d120 2023-03-30 03:56:57,461 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4f2331768da844aca7001af4d006d120, entries=150, sequenceid=145, filesize=11.8 K 2023-03-30 03:56:57,461 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/9c9964df93b74471a0de8200115b8514 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c9964df93b74471a0de8200115b8514 2023-03-30 03:56:57,472 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c9964df93b74471a0de8200115b8514, entries=150, sequenceid=145, filesize=11.8 K 2023-03-30 03:56:57,473 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=100.63 KB/103050 for 08f755044e3e0b03b5ae847b1c94e115 in 887ms, sequenceid=145, compaction requested=false 2023-03-30 03:56:57,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:57,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:57,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:57,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:57,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:57,474 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,474 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:57,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7df3937c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:57,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:57,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:57,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=114.05 KB heapSize=325.22 KB 2023-03-30 03:56:57,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:57,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:57,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:57,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:57,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:57,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:57,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:56:57,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:57,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:57,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:57,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:57,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:57,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:57,479 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:57,482 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:57,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:57,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:57,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:57,483 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:57,484 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,484 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,485 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,487 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,488 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:57,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:57,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,489 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:57,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:57,489 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:57,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:57,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148677504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:57,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148677505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,590 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:57,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:57,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:57,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:57,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:57,592 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:57,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:57,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:57,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:57,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,594 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:57,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:57,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:57,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:57,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:57,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:57,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:57,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:57,596 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,596 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:57,596 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:57,596 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:57,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:57,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:57,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:57,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,597 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:57,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:57,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63d0dd06[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:57,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,599 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,599 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:57,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:57,599 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:57,599 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:57,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,600 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,600 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,600 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:57,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:57,602 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:57,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:57,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:57,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:57,602 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:57,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,604 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:57,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:57,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:57,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,607 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:57,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,607 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:57,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,608 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,608 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:57,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148677609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:57,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148677610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,708 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:57,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:57,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:57,710 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:57,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:57,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:57,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:57,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,711 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,711 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,711 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,711 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:57,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:57,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:57,713 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,713 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:57,713 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:57,713 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:57,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:57,714 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ee4b2f7[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:57,715 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:57,715 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,716 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,716 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:57,716 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:57,716 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:57,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:57,717 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,717 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,717 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,718 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:57,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:57,719 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:57,719 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:57,719 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:57,719 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:57,719 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:57,720 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,720 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,720 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,721 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,722 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:57,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:57,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,724 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:57,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,723 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,724 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:57,725 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:57,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,782 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/ef4f723f9cb749539602fb6c5732beed as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ef4f723f9cb749539602fb6c5732beed 2023-03-30 03:56:57,787 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/A of 08f755044e3e0b03b5ae847b1c94e115 into ef4f723f9cb749539602fb6c5732beed(size=25.5 K), total size for store is 50.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-30 03:56:57,787 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:57,787 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/A, priority=12, startTime=1680148616527; duration=1sec 2023-03-30 03:56:57,787 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:57,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:57,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148677812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:57,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148677812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,825 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:57,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:57,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:57,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:57,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:57,828 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:57,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:57,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:57,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:57,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,829 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,829 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,829 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:57,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:57,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:57,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:57,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:57,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:57,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:57,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:57,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:57,830 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,830 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:57,830 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:57,831 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,831 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:57,831 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:57,831 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,831 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:57,831 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:57,831 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,831 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,832 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,832 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:57,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:57,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7243a148[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:57,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,834 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,834 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:57,834 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:57,834 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:57,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:57,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:57,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:57,838 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:57,838 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:57,838 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:57,838 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:57,838 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:57,838 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,838 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,838 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,839 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,840 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:57,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:57,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,841 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,841 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:57,841 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:57,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:57,908 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330abdb842b08d04fc18e3ee1236bf53b6b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330abdb842b08d04fc18e3ee1236bf53b6b 2023-03-30 03:56:57,942 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:57,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:57,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:57,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:57,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:57,944 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:57,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:57,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:57,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:57,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,946 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:57,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:57,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:57,947 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,947 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:57,947 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:57,947 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:57,947 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:57,948 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,948 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,948 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:57,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13cab40f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:57,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:57,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,950 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,950 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:57,950 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:57,950 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,950 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:57,950 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:57,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,951 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:57,951 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,951 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:57,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:57,952 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:57,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:57,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:57,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:57,953 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:57,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:57,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:57,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,956 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:57,956 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:57,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:57,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:57,956 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:57,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:57,956 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:57,956 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:57,956 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:57,956 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:57,956 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:57,957 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:57,957 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,957 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:57,957 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,056 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:58,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:58,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:58,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:58,059 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:58,060 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,060 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:58,060 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:58,060 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,060 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,061 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:58,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:58,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:58,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:58,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:58,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:58,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:58,063 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,063 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,063 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,063 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:58,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7697739d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:58,064 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:58,064 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:58,065 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,065 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:58,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:58,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:58,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,066 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:58,066 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,066 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:58,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:58,067 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:58,068 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:58,068 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:58,068 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:58,068 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:58,068 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,069 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,070 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,070 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:58,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:58,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:58,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,072 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:58,072 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148678114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148678115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,172 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:58,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:58,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:58,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:58,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:58,175 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:58,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:58,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:58,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,176 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:58,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:58,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:58,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:58,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:58,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:58,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:58,177 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,177 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:58,177 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:58,177 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:58,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:58,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,178 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:58,178 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:58,178 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,179 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,179 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:58,179 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:58,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8f14b2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:58,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,180 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,180 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:58,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:58,180 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:58,180 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:58,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:58,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:58,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:58,182 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:58,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:58,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:58,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:58,182 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:58,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,184 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,185 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,186 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:58,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:58,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,187 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:58,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,187 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:58,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:58,287 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:58,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:58,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:58,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:58,290 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:58,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:58,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:58,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:58,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:58,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:58,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:58,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:58,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:58,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:58,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72a9afaa[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:58,295 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:58,295 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,296 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:58,296 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:58,296 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:58,296 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,296 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:58,297 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,297 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,297 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,297 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:58,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:58,299 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:58,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:58,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:58,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:58,299 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:58,300 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,300 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,300 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,302 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,303 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:58,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:58,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,303 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:58,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,304 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:58,313 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/75db632ea04e42f09ea224897f0036cf 2023-03-30 03:56:58,325 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/6f92798fee7a4d538c3e173fa3a3d2c4 2023-03-30 03:56:58,404 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:58,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:58,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:58,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:58,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:58,407 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:58,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:58,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:58,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,408 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:58,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:58,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:58,409 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,409 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:58,409 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:58,409 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:58,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:58,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,411 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,411 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:58,411 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:58,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3405173c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:58,411 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:58,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:58,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:58,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:58,413 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,413 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,413 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,413 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:58,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:58,414 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:58,414 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:58,414 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:58,414 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:58,414 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:58,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,419 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,420 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:58,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:58,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:58,421 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:58,421 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:58,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,521 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:58,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:58,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:58,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:58,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:58,524 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:58,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:58,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:58,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:58,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:58,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:58,526 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,526 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:58,526 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:58,526 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:58,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:58,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:58,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@794381ba[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:58,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:58,529 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,529 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:58,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,530 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,530 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,530 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:58,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:58,532 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:58,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:58,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:58,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:58,532 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,533 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,533 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,534 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,536 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:58,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:58,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,537 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,537 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,537 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,537 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:58,537 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,537 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:58,537 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,537 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:58,537 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:58,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148678615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:58,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148678619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,637 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:58,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:58,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:58,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:58,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:58,641 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:58,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:58,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:58,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:58,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:58,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:58,644 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:58,644 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,644 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:58,645 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,645 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:58,645 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:58,645 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:58,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c8c20a4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:58,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:58,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,648 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,648 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:58,649 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:58,649 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:58,649 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,649 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:58,650 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,650 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:58,650 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,650 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:58,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:58,653 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:58,653 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:58,653 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:58,653 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:58,653 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:58,654 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,654 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,654 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:58,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,656 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,658 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:58,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:58,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:58,658 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:58,658 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:58,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:58,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:58,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:58,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:58,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/fc1eb5e4d87a43338f7549dd35555ee4 2023-03-30 03:56:58,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/75db632ea04e42f09ea224897f0036cf as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/75db632ea04e42f09ea224897f0036cf 2023-03-30 03:56:58,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/75db632ea04e42f09ea224897f0036cf, entries=250, sequenceid=181, filesize=38.9 K 2023-03-30 03:56:58,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/6f92798fee7a4d538c3e173fa3a3d2c4 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6f92798fee7a4d538c3e173fa3a3d2c4 2023-03-30 03:56:58,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6f92798fee7a4d538c3e173fa3a3d2c4, entries=150, sequenceid=181, filesize=11.8 K 2023-03-30 03:56:58,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/fc1eb5e4d87a43338f7549dd35555ee4 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/fc1eb5e4d87a43338f7549dd35555ee4 2023-03-30 03:56:58,752 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/fc1eb5e4d87a43338f7549dd35555ee4, entries=150, sequenceid=181, filesize=11.8 K 2023-03-30 03:56:58,753 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~325.17 KB/332976, currentSize=67.09 KB/68700 for 08f755044e3e0b03b5ae847b1c94e115 in 1277ms, sequenceid=181, compaction requested=true 2023-03-30 03:56:58,753 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:58,753 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:58,753 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:58,753 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:58,753 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:58,753 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:58,754 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91896 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:58,754 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:58,755 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/A is initiating minor compaction (all files) 2023-03-30 03:56:58,755 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/B is initiating minor compaction (all files) 2023-03-30 03:56:58,755 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/A in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,755 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/B in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,755 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ef4f723f9cb749539602fb6c5732beed, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/9bc713a811c54ce1aa3134b762571533, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/75db632ea04e42f09ea224897f0036cf] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=89.7 K 2023-03-30 03:56:58,755 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4d09419acffc4caa9878056c285b669b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4f2331768da844aca7001af4d006d120, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6f92798fee7a4d538c3e173fa3a3d2c4] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=35.5 K 2023-03-30 03:56:58,755 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting ef4f723f9cb749539602fb6c5732beed, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1680148614936 2023-03-30 03:56:58,755 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4d09419acffc4caa9878056c285b669b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1680148614936 2023-03-30 03:56:58,755 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9bc713a811c54ce1aa3134b762571533, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680148615267 2023-03-30 03:56:58,756 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4f2331768da844aca7001af4d006d120, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680148615267 2023-03-30 03:56:58,756 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 75db632ea04e42f09ea224897f0036cf, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148617465 2023-03-30 03:56:58,756 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6f92798fee7a4d538c3e173fa3a3d2c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148617468 2023-03-30 03:56:58,758 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:58,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:58,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:58,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:58,761 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:58,761 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:58,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:58,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,763 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:58,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:58,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:58,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:58,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:58,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:58,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:58,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:58,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:58,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:58,770 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#B#compaction#163 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:58,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:58,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:58,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:58,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:58,770 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:56:58,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:58,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:58,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:58,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:58,771 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:58,771 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:58,787 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#A#compaction#162 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:59,192 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/d6ca92c04699493bbdcfa22cb560be48 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d6ca92c04699493bbdcfa22cb560be48 2023-03-30 03:56:59,196 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/B of 08f755044e3e0b03b5ae847b1c94e115 into d6ca92c04699493bbdcfa22cb560be48(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:59,196 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:59,196 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/B, priority=13, startTime=1680148618753; duration=0sec 2023-03-30 03:56:59,196 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:56:59,196 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:56:59,197 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:56:59,197 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/C is initiating minor compaction (all files) 2023-03-30 03:56:59,197 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/C in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:59,197 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b3429102086b4beeaa45ee72a3b985eb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c9964df93b74471a0de8200115b8514, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/fc1eb5e4d87a43338f7549dd35555ee4] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=35.5 K 2023-03-30 03:56:59,198 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b3429102086b4beeaa45ee72a3b985eb, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1680148614936 2023-03-30 03:56:59,198 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 9c9964df93b74471a0de8200115b8514, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680148615267 2023-03-30 03:56:59,198 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting fc1eb5e4d87a43338f7549dd35555ee4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148617468 2023-03-30 03:56:59,212 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#C#compaction#165 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:56:59,231 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303303363a8b3e880426295137909069f82a0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303363a8b3e880426295137909069f82a0 2023-03-30 03:56:59,260 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/98639558334345a5b09ad017f45ba49f 2023-03-30 03:56:59,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:59,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:59,648 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/e49ac3e74ff7480ab39d4082cae5701a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e49ac3e74ff7480ab39d4082cae5701a 2023-03-30 03:56:59,649 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/5eaa1055fb2a423993a2aa59eaa7872d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/5eaa1055fb2a423993a2aa59eaa7872d 2023-03-30 03:56:59,653 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/A of 08f755044e3e0b03b5ae847b1c94e115 into e49ac3e74ff7480ab39d4082cae5701a(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:59,653 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:59,653 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/A, priority=13, startTime=1680148618753; duration=0sec 2023-03-30 03:56:59,653 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:59,655 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/C of 08f755044e3e0b03b5ae847b1c94e115 into 5eaa1055fb2a423993a2aa59eaa7872d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:56:59,655 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:59,655 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/C, priority=13, startTime=1680148618753; duration=0sec 2023-03-30 03:56:59,655 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:56:59,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47246 deadline: 1680148679654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47258 deadline: 1680148679656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,670 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/bde6097894f8466aa6b83563bf545742 2023-03-30 03:56:59,679 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/525d6ff9427e4790afb16b7294e65a0b 2023-03-30 03:56:59,683 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/98639558334345a5b09ad017f45ba49f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/98639558334345a5b09ad017f45ba49f 2023-03-30 03:56:59,686 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/98639558334345a5b09ad017f45ba49f, entries=150, sequenceid=195, filesize=25.3 K 2023-03-30 03:56:59,687 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/bde6097894f8466aa6b83563bf545742 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/bde6097894f8466aa6b83563bf545742 2023-03-30 03:56:59,690 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/bde6097894f8466aa6b83563bf545742, entries=150, sequenceid=195, filesize=11.8 K 2023-03-30 03:56:59,692 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/525d6ff9427e4790afb16b7294e65a0b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/525d6ff9427e4790afb16b7294e65a0b 2023-03-30 03:56:59,697 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/525d6ff9427e4790afb16b7294e65a0b, entries=150, sequenceid=195, filesize=11.8 K 2023-03-30 03:56:59,698 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 08f755044e3e0b03b5ae847b1c94e115 in 928ms, sequenceid=195, compaction requested=false 2023-03-30 03:56:59,698 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:56:59,698 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:59,698 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:59,698 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:59,698 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,698 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:59,698 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:59,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,699 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:59,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e71860[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:59,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:59,701 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,701 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,701 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:59,701 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:59,701 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:59,701 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,701 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:59,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:59,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:59,702 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:59,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:59,706 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:59,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:59,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:59,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:59,706 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:59,708 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,708 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,708 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:59,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:59,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:59,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:59,712 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:59,712 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:59,712 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:59,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:59,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:59,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:56:59,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:56:59,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:56:59,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:59,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:56:59,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:59,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:56:59,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:56:59,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148679771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148679772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47184 deadline: 1680148679788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,793 DEBUG [Thread-1005] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:59,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47260 deadline: 1680148679794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,799 DEBUG [Thread-1009] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:59,812 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:59,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:59,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:59,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:59,815 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:59,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:59,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:59,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:59,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,816 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:59,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:59,816 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,816 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,816 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:59,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:59,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:59,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:59,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:59,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:59,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:59,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:59,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:59,817 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:59,817 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:59,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:59,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:59,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:59,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:59,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:59,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:59,818 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:59,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:59,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f4757d7[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:59,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,821 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:59,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:59,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:59,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:59,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:59,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:59,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:59,824 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:59,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:59,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:59,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:59,824 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:59,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:59,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:59,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,826 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:59,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:59,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:59,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:59,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:59,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:59,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:59,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:59,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,828 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:59,829 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:59,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:59,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47216 deadline: 1680148679830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,832 DEBUG [Thread-1007] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:56:59,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148679873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:56:59,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148679874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,929 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:56:59,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:56:59,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:56:59,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:56:59,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:56:59,933 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:56:59,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:59,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:56:59,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:56:59,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,934 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:59,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:59,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:56:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:56:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:56:59,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:56:59,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:56:59,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:56:59,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:56:59,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:56:59,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:56:59,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:56:59,936 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:59,936 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:56:59,936 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:56:59,936 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:56:59,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:56:59,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:56:59,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:56:59,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:56:59,939 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,939 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,939 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,939 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:56:59,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:56:59,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a6c4fd4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:56:59,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,941 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,941 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:56:59,941 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:56:59,941 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:56:59,942 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,942 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:56:59,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:56:59,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:59,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:56:59,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:56:59,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:56:59,944 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:56:59,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:56:59,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:56:59,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:56:59,944 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:56:59,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:56:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:56:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:56:59,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,946 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,946 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:59,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:56:59,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:56:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,947 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:56:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:56:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:56:59,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:56:59,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:56:59,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:56:59,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:56:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:56:59,949 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:56:59,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:56:59,949 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:56:59,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:56:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,049 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,052 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,053 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,053 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,053 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,053 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,054 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,054 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,054 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,054 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,054 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,054 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,054 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,054 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,055 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,055 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2718cd3d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,057 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,057 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,058 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,061 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,061 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:00,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,062 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,064 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,065 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:00,065 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:00,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,065 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:00,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148680077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148680077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,166 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,169 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,170 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,170 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,170 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:00,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,172 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c27e5a6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,174 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,174 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,175 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,175 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,177 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,178 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:00,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,179 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:00,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:00,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:00,181 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:00,181 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:00,181 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,199 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330c505ef666a4d4cbc9617103dee0180eb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c505ef666a4d4cbc9617103dee0180eb 2023-03-30 03:57:00,282 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,285 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,286 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:00,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,288 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,288 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,288 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,288 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39dbe41a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,294 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,294 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,295 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,295 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,295 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,298 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,298 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:00,299 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,299 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,299 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,301 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,304 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,304 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,304 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,304 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,305 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,305 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,305 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,305 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,305 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,305 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:00,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:00,305 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:00,305 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:00,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:00,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148680380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:00,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148680381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,405 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,408 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,409 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:00,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,410 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,410 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,410 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,410 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,411 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,411 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f10e1a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,413 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,413 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,413 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,413 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,413 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,413 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,413 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,413 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,414 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,414 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,414 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,414 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,416 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,416 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:00,417 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,417 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,417 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,417 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,420 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:00,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:00,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,420 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:00,420 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:00,421 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,421 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,421 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,421 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,421 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,421 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,421 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:00,520 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,523 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,524 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:00,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,525 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,525 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,525 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,525 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,527 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60e14a3c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,528 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,528 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,528 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,532 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,532 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:00,533 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,533 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,533 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,533 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,534 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,536 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,536 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:00,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:00,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:00,536 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:00,536 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:00,604 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=218, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/91289384c50b4b2ca3b07ebc479fd8f0 2023-03-30 03:57:00,637 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,639 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,640 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:00,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,641 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,641 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,641 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63688953[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,645 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,645 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,646 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,646 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,647 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,647 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,647 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,647 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,649 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,650 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:00,650 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,650 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,650 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,651 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,652 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:00,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:00,654 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:00,654 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:00,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:00,653 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,653 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,654 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,654 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,754 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,756 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,757 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,757 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,757 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,757 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,757 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,758 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:00,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,759 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,759 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,759 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,759 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,759 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,760 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,760 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,760 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f54a10[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,762 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,762 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,763 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,763 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,763 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,763 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,764 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,764 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,764 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,764 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,766 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,766 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:00,767 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,767 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,767 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:00,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:00,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,770 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,770 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:00,770 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:00,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,871 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,873 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,874 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:00,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,876 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,876 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,876 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,876 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,877 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,877 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,877 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,877 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c7d156a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,879 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,882 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,882 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:00,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:00,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148680883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,884 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:00,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148680884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,885 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,887 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:00,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:00,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:00,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,888 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:00,888 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:00,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:00,988 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:00,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:00,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:00,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:00,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:00,990 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:00,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:00,990 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:00,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,991 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:00,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:00,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:00,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:00,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:00,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:00,993 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,993 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:00,993 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:00,993 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:00,994 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,994 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,995 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,995 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66270e73[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:00,995 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:00,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,996 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:00,996 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:00,996 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:00,996 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:00,997 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,997 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:00,997 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,997 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:00,997 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:00,997 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:00,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:00,999 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:00,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:00,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:00,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:00,999 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:01,000 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,000 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,000 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:01,001 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:01,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:01,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:01,003 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:01,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:01,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:01,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:01,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:01,003 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:01,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,004 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:01,004 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,004 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/b3283b5f9b824fd9b266d8f9e939f069 2023-03-30 03:57:01,104 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:01,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:01,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:01,107 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:01,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:01,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:01,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:01,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,108 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:01,108 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:01,108 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,108 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:01,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:01,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:01,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:01,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:01,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:01,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:01,110 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:01,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@412221f5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:01,111 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:01,111 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,112 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,112 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:01,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:01,112 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:01,112 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:01,113 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:01,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:01,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:01,116 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:01,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:01,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:01,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:01,116 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:01,117 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,117 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:01,117 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:01,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:01,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:01,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:01,120 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:01,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,120 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:01,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:01,220 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:01,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:01,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:01,223 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:01,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:01,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:01,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:01,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,224 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,224 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:01,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:01,224 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,224 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:01,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:01,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:01,225 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:01,225 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:01,225 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:01,225 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:01,225 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:01,225 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,225 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:01,225 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:01,225 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:01,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:01,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:01,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:01,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,226 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,227 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,227 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:01,228 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:01,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e28560b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:01,228 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,228 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,228 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:01,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:01,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:01,229 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,229 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:01,229 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:01,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:01,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:01,231 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:01,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:01,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:01,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:01,231 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:01,232 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,232 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,232 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:01,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:01,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:01,235 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:01,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:01,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:01,236 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:01,236 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:01,336 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:01,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:01,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:01,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:01,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:01,339 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:01,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:01,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:01,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:01,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,340 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:01,340 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:01,340 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,340 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:01,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:01,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:01,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:01,341 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,341 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:01,341 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:01,341 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:01,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:01,342 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,342 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,342 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,342 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:01,344 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:01,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a1ac8e1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:01,344 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,344 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:01,344 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,344 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:01,344 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:01,345 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:01,345 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,345 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:01,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,345 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:01,345 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:01,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:01,347 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:01,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:01,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:01,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:01,347 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:01,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:01,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:01,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,349 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,351 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:01,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:01,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:01,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:01,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:01,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:01,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:01,353 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:01,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,353 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:01,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,353 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:01,353 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/a26b065a99bc44b3aeb893c6bcb5f699 2023-03-30 03:57:01,430 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/91289384c50b4b2ca3b07ebc479fd8f0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91289384c50b4b2ca3b07ebc479fd8f0 2023-03-30 03:57:01,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91289384c50b4b2ca3b07ebc479fd8f0, entries=200, sequenceid=218, filesize=32.1 K 2023-03-30 03:57:01,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/b3283b5f9b824fd9b266d8f9e939f069 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b3283b5f9b824fd9b266d8f9e939f069 2023-03-30 03:57:01,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b3283b5f9b824fd9b266d8f9e939f069, entries=150, sequenceid=218, filesize=11.8 K 2023-03-30 03:57:01,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/a26b065a99bc44b3aeb893c6bcb5f699 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a26b065a99bc44b3aeb893c6bcb5f699 2023-03-30 03:57:01,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a26b065a99bc44b3aeb893c6bcb5f699, entries=150, sequenceid=218, filesize=11.8 K 2023-03-30 03:57:01,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 08f755044e3e0b03b5ae847b1c94e115 in 1684ms, sequenceid=218, compaction requested=true 2023-03-30 03:57:01,443 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:01,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:01,443 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:57:01,443 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:57:01,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:01,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:01,444 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:57:01,445 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/B is initiating minor compaction (all files) 2023-03-30 03:57:01,445 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:57:01,445 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/B in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,445 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/A is initiating minor compaction (all files) 2023-03-30 03:57:01,445 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d6ca92c04699493bbdcfa22cb560be48, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/bde6097894f8466aa6b83563bf545742, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b3283b5f9b824fd9b266d8f9e939f069] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=35.6 K 2023-03-30 03:57:01,445 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/A in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,445 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e49ac3e74ff7480ab39d4082cae5701a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/98639558334345a5b09ad017f45ba49f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91289384c50b4b2ca3b07ebc479fd8f0] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=83.1 K 2023-03-30 03:57:01,445 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d6ca92c04699493bbdcfa22cb560be48, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148617468 2023-03-30 03:57:01,445 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting e49ac3e74ff7480ab39d4082cae5701a, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148617468 2023-03-30 03:57:01,445 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting bde6097894f8466aa6b83563bf545742, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680148617490 2023-03-30 03:57:01,445 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 98639558334345a5b09ad017f45ba49f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680148617490 2023-03-30 03:57:01,446 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b3283b5f9b824fd9b266d8f9e939f069, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148619649 2023-03-30 03:57:01,446 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 91289384c50b4b2ca3b07ebc479fd8f0, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148619645 2023-03-30 03:57:01,453 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#B#compaction#171 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:01,454 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:01,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:01,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:01,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:01,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:01,457 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:01,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:01,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:01,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:01,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,458 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:01,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:01,458 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,458 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:01,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:01,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:01,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:01,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:01,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:01,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:01,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:01,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:01,471 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#A#compaction#172 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:01,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:01,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:01,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:01,472 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:57:01,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:01,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:01,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:01,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:01,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:01,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:01,482 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/f82ae02d71124fad8c8594fe11cf8714 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/f82ae02d71124fad8c8594fe11cf8714 2023-03-30 03:57:01,502 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/B of 08f755044e3e0b03b5ae847b1c94e115 into f82ae02d71124fad8c8594fe11cf8714(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:01,502 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:01,502 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/B, priority=13, startTime=1680148621443; duration=0sec 2023-03-30 03:57:01,502 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:01,502 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:57:01,511 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:57:01,511 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/C is initiating minor compaction (all files) 2023-03-30 03:57:01,511 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/C in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:01,511 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/5eaa1055fb2a423993a2aa59eaa7872d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/525d6ff9427e4790afb16b7294e65a0b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a26b065a99bc44b3aeb893c6bcb5f699] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=35.6 K 2023-03-30 03:57:01,511 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5eaa1055fb2a423993a2aa59eaa7872d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680148617468 2023-03-30 03:57:01,513 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 525d6ff9427e4790afb16b7294e65a0b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680148617490 2023-03-30 03:57:01,513 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a26b065a99bc44b3aeb893c6bcb5f699, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148619649 2023-03-30 03:57:01,539 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#C#compaction#174 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:01,556 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/2c0fcc8395db460bb3a45b9aaea1b8d2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/2c0fcc8395db460bb3a45b9aaea1b8d2 2023-03-30 03:57:01,563 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/C of 08f755044e3e0b03b5ae847b1c94e115 into 2c0fcc8395db460bb3a45b9aaea1b8d2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:01,563 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:01,563 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/C, priority=13, startTime=1680148621444; duration=0sec 2023-03-30 03:57:01,563 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:01,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:01,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:01,908 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033052690c91ab6349fca4e7832a2c214a53 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033052690c91ab6349fca4e7832a2c214a53 2023-03-30 03:57:01,926 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/decc2ab455014630bbda3aa8bbd35884 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/decc2ab455014630bbda3aa8bbd35884 2023-03-30 03:57:01,930 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/A of 08f755044e3e0b03b5ae847b1c94e115 into decc2ab455014630bbda3aa8bbd35884(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:01,930 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:01,930 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/A, priority=13, startTime=1680148621443; duration=0sec 2023-03-30 03:57:01,930 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:01,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148681930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:01,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148681930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:02,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148682035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:02,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148682038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:02,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148682240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:02,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148682240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,321 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/91848ccddcae425c80c3f5f3b42fc1dd 2023-03-30 03:57:02,368 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/a0b7ae4fa3ca464fbf07d71fc9eabd4c 2023-03-30 03:57:02,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148682544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148682545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,793 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/d2cd9ba882d840d08346a5ae48c72559 2023-03-30 03:57:02,798 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/91848ccddcae425c80c3f5f3b42fc1dd as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91848ccddcae425c80c3f5f3b42fc1dd 2023-03-30 03:57:02,804 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91848ccddcae425c80c3f5f3b42fc1dd, entries=150, sequenceid=230, filesize=25.3 K 2023-03-30 03:57:02,805 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/a0b7ae4fa3ca464fbf07d71fc9eabd4c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a0b7ae4fa3ca464fbf07d71fc9eabd4c 2023-03-30 03:57:02,811 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a0b7ae4fa3ca464fbf07d71fc9eabd4c, entries=150, sequenceid=230, filesize=11.8 K 2023-03-30 03:57:02,812 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/d2cd9ba882d840d08346a5ae48c72559 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d2cd9ba882d840d08346a5ae48c72559 2023-03-30 03:57:02,817 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d2cd9ba882d840d08346a5ae48c72559, entries=150, sequenceid=230, filesize=11.8 K 2023-03-30 03:57:02,818 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 08f755044e3e0b03b5ae847b1c94e115 in 1346ms, sequenceid=230, compaction requested=false 2023-03-30 03:57:02,819 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:02,819 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:02,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:02,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:02,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:02,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:02,820 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:02,820 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:02,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:02,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:02,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:02,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54901b47[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:02,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:02,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:02,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:02,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:02,827 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:02,827 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:02,827 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:02,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:02,827 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:02,828 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:02,828 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:02,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:02,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:02,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:02,831 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:02,831 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:02,831 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:02,831 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:02,831 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:02,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:02,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:02,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,849 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:02,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:02,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:02,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:02,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:02,850 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:02,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:02,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:02,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:02,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:02,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:02,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,850 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:02,850 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:02,951 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:02,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:02,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:02,960 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:02,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:02,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:02,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:02,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:02,961 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,961 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:02,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:02,962 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:02,962 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:02,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:02,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:02,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:02,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:02,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:02,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:02,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:02,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:02,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:02,964 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:02,966 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:02,968 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:57:02,968 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:02,969 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:02,969 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:02,969 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:02,969 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:02,969 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:03,033 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033050abb6cf0f2f4682a13d8b242f4c09f1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033050abb6cf0f2f4682a13d8b242f4c09f1 2023-03-30 03:57:03,051 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:03,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:03,063 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=253, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/eb56c3348f09459ab1d323ca6e420e0c 2023-03-30 03:57:03,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148683072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148683074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,105 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/5745e69e63aa4158b404579251a96a15 2023-03-30 03:57:03,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148683179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148683182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148683384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148683384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,559 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/a0c4a521f872469382b26ed937b7c133 2023-03-30 03:57:03,565 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/eb56c3348f09459ab1d323ca6e420e0c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/eb56c3348f09459ab1d323ca6e420e0c 2023-03-30 03:57:03,575 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/eb56c3348f09459ab1d323ca6e420e0c, entries=150, sequenceid=253, filesize=25.3 K 2023-03-30 03:57:03,576 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/5745e69e63aa4158b404579251a96a15 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5745e69e63aa4158b404579251a96a15 2023-03-30 03:57:03,584 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5745e69e63aa4158b404579251a96a15, entries=150, sequenceid=253, filesize=11.8 K 2023-03-30 03:57:03,585 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/a0c4a521f872469382b26ed937b7c133 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a0c4a521f872469382b26ed937b7c133 2023-03-30 03:57:03,597 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a0c4a521f872469382b26ed937b7c133, entries=150, sequenceid=253, filesize=11.8 K 2023-03-30 03:57:03,598 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 08f755044e3e0b03b5ae847b1c94e115 in 629ms, sequenceid=253, compaction requested=true 2023-03-30 03:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:03,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:03,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:03,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:03,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:03,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46db5a9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:03,604 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:03,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,605 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:03,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:03,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:03,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:03,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:03,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:03,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:03,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:03,607 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:03,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:03,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:03,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:03,607 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:03,608 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,608 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:03,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,608 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:03,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:03,609 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:03,615 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:03,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:03,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:03,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:03,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:03,625 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:03,624 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:03,625 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,625 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:03,625 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:03,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:03,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:57:03,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:03,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:03,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:03,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:03,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:03,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:03,702 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330517ee75931d94b949e62e63d31e9cf5d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330517ee75931d94b949e62e63d31e9cf5d 2023-03-30 03:57:03,725 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:03,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:03,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:03,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:03,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:03,728 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:03,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:03,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:03,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:03,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,729 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:03,729 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:03,729 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,729 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:03,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:03,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:03,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:03,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:03,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:03,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:03,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:03,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:03,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:03,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,731 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:03,732 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,732 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:03,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d93321[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:03,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,734 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,734 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:03,734 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:03,734 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:03,735 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:03,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,735 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:03,735 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:03,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:03,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:03,735 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:03,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:03,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:03,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:03,735 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:03,736 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,736 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:03,736 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:03,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,737 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,738 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:03,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:03,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:03,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:03,740 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:03,740 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,740 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148683745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148683745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,841 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:03,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:03,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:03,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148683848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:03,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148683848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:03,854 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:03,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:03,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:03,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:03,855 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,855 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,857 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:03,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:03,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:03,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:03,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:03,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:03,869 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:03,869 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:03,869 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:03,869 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:03,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:03,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,875 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:03,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:03,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:03,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d8bb1b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:03,877 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:03,877 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:03,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:03,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:03,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,879 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:03,879 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:03,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:03,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:03,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:03,881 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:03,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:03,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:03,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:03,882 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:03,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:03,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,884 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:03,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,885 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:03,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:03,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,888 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:03,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:03,888 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:03,889 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:03,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:03,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:03,989 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:03,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:03,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:03,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:03,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:03,991 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:03,991 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:03,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:03,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:03,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,993 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:03,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:03,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:03,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:03,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:03,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:03,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:03,995 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:03,995 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:03,995 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:03,995 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:03,995 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:03,995 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:03,995 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:03,996 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:03,997 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:03,997 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:03,997 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:03,997 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,997 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:03,997 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:03,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:03,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:03,998 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:03,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:03,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2046cce4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:03,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:03,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,000 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:04,000 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,000 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:04,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:04,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:04,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:04,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:04,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:04,003 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:04,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:04,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:04,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:04,004 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:04,005 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,005 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,005 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,005 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,009 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,022 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,022 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,023 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:04,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:04,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,024 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:04,024 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:04,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:04,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:04,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:04,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148684051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148684051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,112 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=266, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/bec961f4db6047e4a6eec8d5da6393b9 2023-03-30 03:57:04,124 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:04,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:04,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:04,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:04,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:04,129 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:04,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:04,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:04,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,130 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:04,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:04,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:04,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:04,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:04,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:04,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:04,133 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,133 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:04,133 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:04,133 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:04,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:04,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:04,133 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:04,134 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,134 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:04,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496ff5b8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:04,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:04,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,137 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:04,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:04,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:04,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:04,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:04,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,138 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:04,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:04,140 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:04,140 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:04,140 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:04,140 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:04,140 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:04,142 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,142 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,142 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,143 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,144 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,147 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,147 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,147 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,147 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,147 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:04,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:04,147 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,147 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,147 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,147 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:04,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,148 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:04,148 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:04,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,248 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:04,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:04,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:04,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:04,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:04,254 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:04,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:04,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:04,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,255 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:04,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:04,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:04,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:04,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:04,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:04,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:04,258 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,258 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:04,258 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:04,258 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:04,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:04,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:04,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:04,259 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:04,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@284e978f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:04,260 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:04,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,261 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,261 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:04,261 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:04,261 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:04,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:04,262 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:04,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:04,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:04,264 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:04,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:04,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:04,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:04,264 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:04,265 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,265 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,265 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,266 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,267 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,268 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:04,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:04,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,269 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,269 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:04,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:04,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:04,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148684357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:04,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148684357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,369 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:04,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:04,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:04,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:04,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:04,371 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:04,371 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,371 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:04,371 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:04,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,372 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:04,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:04,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:04,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:04,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:04,373 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:04,373 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,373 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:04,373 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:04,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:04,374 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,374 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:04,374 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:04,374 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:04,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:04,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:04,374 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:04,375 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:04,376 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:04,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@197fe1fe[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:04,376 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,376 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,377 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:04,377 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:04,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:04,377 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:04,377 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,377 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,377 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,377 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:04,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:04,379 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:04,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:04,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:04,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:04,379 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:04,380 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,380 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,380 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,380 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:04,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:04,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,385 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:04,385 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:04,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,485 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:04,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:04,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:04,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:04,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:04,488 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:04,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:04,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:04,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:04,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:04,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:04,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:04,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:04,490 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,490 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:04,490 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:04,490 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:04,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:04,491 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:04,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:04,492 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:04,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f483fd9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:04,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,493 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,493 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:04,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:04,493 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:04,493 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:04,494 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:04,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,494 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:04,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:04,496 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:04,496 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:04,496 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:04,496 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:04,496 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:04,497 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,497 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,497 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:04,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:04,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,499 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,499 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:04,499 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:04,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,500 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:04,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,500 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,500 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,500 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,500 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/af7bd47377f148f3a7599540c47fc9ae 2023-03-30 03:57:04,600 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:04,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:04,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:04,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:04,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:04,602 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:04,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:04,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:04,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,604 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,604 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,604 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:04,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:04,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:04,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:04,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:04,605 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,605 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:04,605 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:04,605 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:04,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:04,606 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:04,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393cd9bd[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:04,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:04,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,608 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:04,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:04,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:04,608 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,608 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:04,610 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,610 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:04,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,610 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,610 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:04,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:04,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:04,612 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:04,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:04,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:04,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:04,612 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:04,612 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,613 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,613 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,614 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,615 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:04,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:04,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,615 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:04,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,615 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:04,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,716 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:04,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:04,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:04,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:04,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:04,718 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:04,718 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,718 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:04,718 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:04,719 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,719 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,719 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,719 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,720 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,720 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:04,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:04,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:04,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:04,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:04,720 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:04,720 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:04,721 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,721 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:04,721 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:04,721 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:04,722 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,722 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,722 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,722 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:04,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6512c3fa[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:04,723 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:04,723 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,723 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:04,723 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:04,723 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:04,723 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:04,724 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:04,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:04,726 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:04,726 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:04,726 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:04,726 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:04,726 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:04,726 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,726 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,727 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,729 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,729 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,729 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,729 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,729 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,729 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,729 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,729 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:04,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:04,729 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:04,729 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,734 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:04,830 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:04,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:04,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:04,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:04,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:04,832 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:04,832 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,832 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:04,832 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:04,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,834 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:04,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:04,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:04,835 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,835 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:04,835 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:04,835 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:04,835 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:04,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,836 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:04,837 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:04,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@304beb7[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:04,837 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,838 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,838 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:04,838 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:04,838 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:04,838 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,838 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:04,838 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,838 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:04,838 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,838 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:04,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:04,840 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:04,840 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:04,840 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:04,840 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:04,840 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:04,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:04,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,841 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:04,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:04,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:04,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:04,844 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:04,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,844 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:04,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:04,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148684863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:04,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148684863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/27c5ae665fdb49baa75553c845d782c5 2023-03-30 03:57:04,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/bec961f4db6047e4a6eec8d5da6393b9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bec961f4db6047e4a6eec8d5da6393b9 2023-03-30 03:57:04,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bec961f4db6047e4a6eec8d5da6393b9, entries=200, sequenceid=266, filesize=32.3 K 2023-03-30 03:57:04,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/af7bd47377f148f3a7599540c47fc9ae as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/af7bd47377f148f3a7599540c47fc9ae 2023-03-30 03:57:04,944 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/af7bd47377f148f3a7599540c47fc9ae, entries=150, sequenceid=266, filesize=11.9 K 2023-03-30 03:57:04,944 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:04,945 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/27c5ae665fdb49baa75553c845d782c5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/27c5ae665fdb49baa75553c845d782c5 2023-03-30 03:57:04,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:04,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:04,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:04,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:04,948 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:04,948 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,948 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:04,948 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:04,948 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,951 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/27c5ae665fdb49baa75553c845d782c5, entries=150, sequenceid=266, filesize=11.9 K 2023-03-30 03:57:04,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 08f755044e3e0b03b5ae847b1c94e115 in 1265ms, sequenceid=266, compaction requested=true 2023-03-30 03:57:04,952 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:04,952 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:04,952 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:04,952 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:04,952 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:04,952 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:04,954 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48722 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:04,954 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/B is initiating minor compaction (all files) 2023-03-30 03:57:04,954 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111202 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:04,954 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/B in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,954 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/A is initiating minor compaction (all files) 2023-03-30 03:57:04,954 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/A in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,954 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/f82ae02d71124fad8c8594fe11cf8714, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a0b7ae4fa3ca464fbf07d71fc9eabd4c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5745e69e63aa4158b404579251a96a15, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/af7bd47377f148f3a7599540c47fc9ae] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=47.6 K 2023-03-30 03:57:04,954 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/decc2ab455014630bbda3aa8bbd35884, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91848ccddcae425c80c3f5f3b42fc1dd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/eb56c3348f09459ab1d323ca6e420e0c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bec961f4db6047e4a6eec8d5da6393b9] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=108.6 K 2023-03-30 03:57:04,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,954 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:04,954 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting f82ae02d71124fad8c8594fe11cf8714, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148619649 2023-03-30 03:57:04,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:04,955 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting decc2ab455014630bbda3aa8bbd35884, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148619649 2023-03-30 03:57:04,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:04,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,955 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a0b7ae4fa3ca464fbf07d71fc9eabd4c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680148619765 2023-03-30 03:57:04,955 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 91848ccddcae425c80c3f5f3b42fc1dd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680148619765 2023-03-30 03:57:04,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:04,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:04,956 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5745e69e63aa4158b404579251a96a15, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680148621904 2023-03-30 03:57:04,956 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting eb56c3348f09459ab1d323ca6e420e0c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680148621904 2023-03-30 03:57:04,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:04,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:04,956 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting bec961f4db6047e4a6eec8d5da6393b9, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680148623049 2023-03-30 03:57:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:04,956 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting af7bd47377f148f3a7599540c47fc9ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680148623049 2023-03-30 03:57:04,956 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:04,957 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:04,957 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 03:57:04,957 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:04,957 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:04,957 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:04,957 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:04,957 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:04,958 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:04,976 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#B#compaction#185 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:04,979 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#A#compaction#183 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:04,997 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330dea487304a7944c6b146f26b32971d64 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dea487304a7944c6b146f26b32971d64 2023-03-30 03:57:05,002 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=285, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/fe64b3e7d51a4f5b9838e85539c6f95c 2023-03-30 03:57:05,391 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/5bd78040882d487dbab796f866fe2731 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5bd78040882d487dbab796f866fe2731 2023-03-30 03:57:05,395 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/B of 08f755044e3e0b03b5ae847b1c94e115 into 5bd78040882d487dbab796f866fe2731(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:05,395 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:05,395 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/B, priority=12, startTime=1680148624952; duration=0sec 2023-03-30 03:57:05,395 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:05,395 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:05,397 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48722 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:05,397 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/C is initiating minor compaction (all files) 2023-03-30 03:57:05,397 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/C in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:05,397 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/2c0fcc8395db460bb3a45b9aaea1b8d2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d2cd9ba882d840d08346a5ae48c72559, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a0c4a521f872469382b26ed937b7c133, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/27c5ae665fdb49baa75553c845d782c5] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=47.6 K 2023-03-30 03:57:05,398 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2c0fcc8395db460bb3a45b9aaea1b8d2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680148619649 2023-03-30 03:57:05,398 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d2cd9ba882d840d08346a5ae48c72559, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680148619765 2023-03-30 03:57:05,398 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a0c4a521f872469382b26ed937b7c133, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680148621904 2023-03-30 03:57:05,399 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 27c5ae665fdb49baa75553c845d782c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680148623049 2023-03-30 03:57:05,411 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/79d4cce3eca5445899a42d06995512de as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/79d4cce3eca5445899a42d06995512de 2023-03-30 03:57:05,417 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/A of 08f755044e3e0b03b5ae847b1c94e115 into 79d4cce3eca5445899a42d06995512de(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:05,417 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:05,417 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/A, priority=12, startTime=1680148624952; duration=0sec 2023-03-30 03:57:05,417 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:05,422 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/dd02596f03db46b390515db8bfdfb9ca 2023-03-30 03:57:05,423 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#C#compaction#187 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:05,842 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/b945a87ea6a841368c55f62cc55b311e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b945a87ea6a841368c55f62cc55b311e 2023-03-30 03:57:05,842 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/df5a79c49abb443ab061988b8194c0e1 2023-03-30 03:57:05,846 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/fe64b3e7d51a4f5b9838e85539c6f95c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/fe64b3e7d51a4f5b9838e85539c6f95c 2023-03-30 03:57:05,846 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/C of 08f755044e3e0b03b5ae847b1c94e115 into b945a87ea6a841368c55f62cc55b311e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:05,846 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:05,846 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/C, priority=12, startTime=1680148624952; duration=0sec 2023-03-30 03:57:05,846 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:05,849 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/fe64b3e7d51a4f5b9838e85539c6f95c, entries=150, sequenceid=285, filesize=25.4 K 2023-03-30 03:57:05,850 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/dd02596f03db46b390515db8bfdfb9ca as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dd02596f03db46b390515db8bfdfb9ca 2023-03-30 03:57:05,854 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dd02596f03db46b390515db8bfdfb9ca, entries=150, sequenceid=285, filesize=11.9 K 2023-03-30 03:57:05,855 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/df5a79c49abb443ab061988b8194c0e1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df5a79c49abb443ab061988b8194c0e1 2023-03-30 03:57:05,859 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df5a79c49abb443ab061988b8194c0e1, entries=150, sequenceid=285, filesize=11.9 K 2023-03-30 03:57:05,859 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 08f755044e3e0b03b5ae847b1c94e115 in 902ms, sequenceid=285, compaction requested=false 2023-03-30 03:57:05,860 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:05,860 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:05,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:05,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:05,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:05,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:05,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,861 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:05,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:05,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:05,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d7ceff9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:05,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:05,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:05,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:05,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:05,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:05,866 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:05,868 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:05,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:05,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:05,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:05,869 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:05,870 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:05,871 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:05,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,871 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:05,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:05,872 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:05,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:05,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:05,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:05,875 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:05,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,875 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:05,875 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:05,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:05,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:57:05,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:05,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:05,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:05,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:05,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:05,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:05,958 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033067c1aaafa508421f8f785b3c4f17c381 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033067c1aaafa508421f8f785b3c4f17c381 2023-03-30 03:57:05,976 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:05,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:05,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148625970, max timestamp=1680148625975Num uniques 859; 2023-03-30 03:57:05,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:05,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148625970, max timestamp=1680148625975Num uniques 859; 2023-03-30 03:57:05,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148625970, max timestamp=1680148625975Num uniques 859; 2023-03-30 03:57:05,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:05,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:05,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:05,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:05,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:05,983 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:05,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:05,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:05,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:05,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:05,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:05,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:05,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:05,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:05,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:05,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:05,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:05,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:05,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:05,990 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:05,990 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:05,990 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:05,990 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:05,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:05,999 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:06,001 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:06,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125226da[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:06,001 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,002 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:06,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:06,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:06,002 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:06,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,003 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:06,003 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:06,003 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:06,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:06,004 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:06,005 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:06,005 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:06,005 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:06,005 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:06,006 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,006 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,006 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148686006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:06,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:06,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:06,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:06,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:06,010 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:06,010 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:06,009 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:06,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:06,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148686006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,110 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:06,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148686109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:06,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:06,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148686111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:06,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:06,112 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:06,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:06,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:06,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:06,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,113 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:06,113 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:06,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:06,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:06,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:06,115 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,115 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:06,115 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:06,115 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:06,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:06,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@612c2a79[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:06,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:06,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,118 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:06,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:06,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:06,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:06,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:06,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:06,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:06,121 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:06,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:06,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:06,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:06,121 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:06,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,122 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:06,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,122 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:06,124 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:06,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:06,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:06,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:06,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:06,124 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:06,124 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:06,124 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:06,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:06,225 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:06,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:06,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:06,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:06,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:06,227 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:06,227 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:06,227 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:06,227 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:06,227 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,228 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,228 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:06,228 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:06,228 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,228 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:06,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:06,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:06,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:06,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:06,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:06,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:06,229 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:06,229 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,230 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:06,230 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:06,230 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:06,231 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,231 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,231 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,231 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477bc1ed[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:06,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:06,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,233 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:06,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:06,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:06,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:06,234 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:06,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:06,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:06,236 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:06,236 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:06,236 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:06,236 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:06,236 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:06,236 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,236 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,237 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,237 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,238 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,256 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,256 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:06,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:06,256 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:06,257 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:06,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:06,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:06,256 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:06,257 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:06,257 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,257 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:06,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:06,257 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:06,257 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,257 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148686311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148686314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,357 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:06,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:06,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:06,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:06,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:06,360 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:06,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:06,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:06,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:06,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,361 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:06,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:06,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:06,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:06,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:06,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:06,362 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,362 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:06,362 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:06,362 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:06,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:06,363 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,363 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,364 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,364 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:06,364 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=298, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/ad54ccc683f940c4ba15f8e74d5ba103 2023-03-30 03:57:06,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:06,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd2eddc[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:06,365 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:06,365 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,366 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:06,366 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:06,366 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:06,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:06,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:06,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:06,367 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:06,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:06,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:06,372 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:06,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:06,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:06,372 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:06,372 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:06,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/34caa667579b450793dacc0de1f6e80e 2023-03-30 03:57:06,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:06,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:06,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/db0219d577f84f22b20d5c63d0b042f5 2023-03-30 03:57:06,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:06,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:06,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,385 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/ad54ccc683f940c4ba15f8e74d5ba103 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ad54ccc683f940c4ba15f8e74d5ba103 2023-03-30 03:57:06,391 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,391 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,391 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:06,391 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:06,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:06,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:06,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:06,391 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:06,391 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:06,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:06,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ad54ccc683f940c4ba15f8e74d5ba103, entries=350, sequenceid=298, filesize=52.9 K 2023-03-30 03:57:06,391 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:06,391 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,391 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:06,392 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,392 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:06,392 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/34caa667579b450793dacc0de1f6e80e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/34caa667579b450793dacc0de1f6e80e 2023-03-30 03:57:06,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/34caa667579b450793dacc0de1f6e80e, entries=100, sequenceid=298, filesize=9.5 K 2023-03-30 03:57:06,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/db0219d577f84f22b20d5c63d0b042f5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/db0219d577f84f22b20d5c63d0b042f5 2023-03-30 03:57:06,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/db0219d577f84f22b20d5c63d0b042f5, entries=100, sequenceid=298, filesize=9.5 K 2023-03-30 03:57:06,404 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 08f755044e3e0b03b5ae847b1c94e115 in 467ms, sequenceid=298, compaction requested=true 2023-03-30 03:57:06,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:06,404 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:57:06,404 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:06,404 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:06,405 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:57:06,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:06,405 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:57:06,405 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/A is initiating minor compaction (all files) 2023-03-30 03:57:06,405 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:57:06,405 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/A in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,405 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/B is initiating minor compaction (all files) 2023-03-30 03:57:06,405 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/B in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,405 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/79d4cce3eca5445899a42d06995512de, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/fe64b3e7d51a4f5b9838e85539c6f95c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ad54ccc683f940c4ba15f8e74d5ba103] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=104.4 K 2023-03-30 03:57:06,406 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5bd78040882d487dbab796f866fe2731, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dd02596f03db46b390515db8bfdfb9ca, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/34caa667579b450793dacc0de1f6e80e] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=33.9 K 2023-03-30 03:57:06,406 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 79d4cce3eca5445899a42d06995512de, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680148623049 2023-03-30 03:57:06,406 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5bd78040882d487dbab796f866fe2731, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680148623049 2023-03-30 03:57:06,406 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting fe64b3e7d51a4f5b9838e85539c6f95c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680148623739 2023-03-30 03:57:06,406 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting dd02596f03db46b390515db8bfdfb9ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680148623739 2023-03-30 03:57:06,406 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting ad54ccc683f940c4ba15f8e74d5ba103, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148625883 2023-03-30 03:57:06,406 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 34caa667579b450793dacc0de1f6e80e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148625933 2023-03-30 03:57:06,412 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#B#compaction#193 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:06,419 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#A#compaction#192 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:06,424 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/ac51d5c9fdff44368f9f5a4d297f84b6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/ac51d5c9fdff44368f9f5a4d297f84b6 2023-03-30 03:57:06,429 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/B of 08f755044e3e0b03b5ae847b1c94e115 into ac51d5c9fdff44368f9f5a4d297f84b6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:06,430 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:06,430 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/B, priority=13, startTime=1680148626404; duration=0sec 2023-03-30 03:57:06,430 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:06,430 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:57:06,431 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:57:06,431 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/C is initiating minor compaction (all files) 2023-03-30 03:57:06,431 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/C in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,431 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b945a87ea6a841368c55f62cc55b311e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df5a79c49abb443ab061988b8194c0e1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/db0219d577f84f22b20d5c63d0b042f5] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=33.9 K 2023-03-30 03:57:06,432 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b945a87ea6a841368c55f62cc55b311e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680148623049 2023-03-30 03:57:06,432 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting df5a79c49abb443ab061988b8194c0e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680148623739 2023-03-30 03:57:06,432 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting db0219d577f84f22b20d5c63d0b042f5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148625933 2023-03-30 03:57:06,439 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#C#compaction#194 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:06,492 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:06,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:06,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:06,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:06,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:06,494 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:06,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:06,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:06,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:06,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,495 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:06,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:06,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:06,496 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,496 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:06,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:06,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:06,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:06,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:06,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:06,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:06,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:06,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:06,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:06,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:06,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:06,497 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-30 03:57:06,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:06,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:06,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:06,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:06,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:06,498 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:06,514 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033094f86564ac3d4ec6a37750cf012c2c9d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033094f86564ac3d4ec6a37750cf012c2c9d 2023-03-30 03:57:06,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:06,618 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:06,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148686627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148686628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148686729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148686730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,858 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/075ce03895504dc8b29c456eb0378dc0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/075ce03895504dc8b29c456eb0378dc0 2023-03-30 03:57:06,862 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/C of 08f755044e3e0b03b5ae847b1c94e115 into 075ce03895504dc8b29c456eb0378dc0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:06,862 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:06,862 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/C, priority=13, startTime=1680148626405; duration=0sec 2023-03-30 03:57:06,862 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:06,919 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/2e2610955bf0469e97954d0a62bbf20c 2023-03-30 03:57:06,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148686932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:06,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:06,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148686933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:07,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148687237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:07,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148687239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,329 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/187b4447219e4d3f936008fa7f6fdfe2 2023-03-30 03:57:07,648 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/1d41adaece1c4118a4886d81e167c9d5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/1d41adaece1c4118a4886d81e167c9d5 2023-03-30 03:57:07,652 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/A of 08f755044e3e0b03b5ae847b1c94e115 into 1d41adaece1c4118a4886d81e167c9d5(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-30 03:57:07,652 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:07,652 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/A, priority=13, startTime=1680148626404; duration=1sec 2023-03-30 03:57:07,652 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:07,737 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/51bbbe1bce274d1280d98d60e7d43049 2023-03-30 03:57:07,742 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/2e2610955bf0469e97954d0a62bbf20c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/2e2610955bf0469e97954d0a62bbf20c 2023-03-30 03:57:07,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:07,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148687741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:07,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148687742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,745 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/2e2610955bf0469e97954d0a62bbf20c, entries=150, sequenceid=335, filesize=25.4 K 2023-03-30 03:57:07,746 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/187b4447219e4d3f936008fa7f6fdfe2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/187b4447219e4d3f936008fa7f6fdfe2 2023-03-30 03:57:07,750 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/187b4447219e4d3f936008fa7f6fdfe2, entries=150, sequenceid=335, filesize=11.9 K 2023-03-30 03:57:07,751 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/51bbbe1bce274d1280d98d60e7d43049 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/51bbbe1bce274d1280d98d60e7d43049 2023-03-30 03:57:07,755 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/51bbbe1bce274d1280d98d60e7d43049, entries=150, sequenceid=335, filesize=11.9 K 2023-03-30 03:57:07,756 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=46.96 KB/48090 for 08f755044e3e0b03b5ae847b1c94e115 in 1259ms, sequenceid=335, compaction requested=false 2023-03-30 03:57:07,756 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:07,756 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:07,756 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:07,756 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:07,756 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,756 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:07,756 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:07,757 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:07,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:07,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,758 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:07,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:07,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eb5624a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:07,760 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:07,760 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:07,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:07,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:07,760 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:07,760 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:07,760 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:07,760 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,761 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:07,761 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:07,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:07,764 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:07,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:07,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:07,764 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:07,764 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:07,765 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,765 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,765 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:07,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:07,765 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:07,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,768 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:07,768 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:07,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:07,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:07,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:07,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:07,768 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:07,768 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:07,768 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:07,769 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:07,768 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:07,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:07,769 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,769 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:07,769 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:07,769 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,869 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:07,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:07,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:07,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:07,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:07,874 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:07,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:07,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:07,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:07,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:07,875 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:07,875 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:07,875 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:07,876 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:07,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:07,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:07,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:07,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:07,877 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:07,877 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:07,877 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:07,877 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:07,877 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:07,877 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:07,877 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:07,877 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:57:07,877 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:07,877 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:07,878 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:07,878 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:07,878 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:07,878 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:07,895 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d58a605de58f4381b3ef1540f7c6a911 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d58a605de58f4381b3ef1540f7c6a911 2023-03-30 03:57:07,909 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/5fc86cfcec4b4f6a902e1384186f426d 2023-03-30 03:57:08,324 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/dafc03b0ba37406b9b7307de357c206b 2023-03-30 03:57:08,748 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/8167d4b9221f4a708dc8ead8741c3066 2023-03-30 03:57:08,753 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/5fc86cfcec4b4f6a902e1384186f426d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/5fc86cfcec4b4f6a902e1384186f426d 2023-03-30 03:57:08,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:08,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:08,759 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/5fc86cfcec4b4f6a902e1384186f426d, entries=100, sequenceid=347, filesize=18.6 K 2023-03-30 03:57:08,760 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/dafc03b0ba37406b9b7307de357c206b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dafc03b0ba37406b9b7307de357c206b 2023-03-30 03:57:08,763 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dafc03b0ba37406b9b7307de357c206b, entries=100, sequenceid=347, filesize=9.5 K 2023-03-30 03:57:08,764 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/8167d4b9221f4a708dc8ead8741c3066 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/8167d4b9221f4a708dc8ead8741c3066 2023-03-30 03:57:08,768 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/8167d4b9221f4a708dc8ead8741c3066, entries=100, sequenceid=347, filesize=9.5 K 2023-03-30 03:57:08,770 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=13.42 KB/13740 for 08f755044e3e0b03b5ae847b1c94e115 in 893ms, sequenceid=347, compaction requested=true 2023-03-30 03:57:08,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:08,770 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:08,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:08,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:08,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:08,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:08,771 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:08,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:08,772 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,772 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:08,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:08,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:08,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:08,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e4ae207[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:08,773 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:08,773 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:08,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:08,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:08,774 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,774 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:08,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:08,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:08,775 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:08,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:08,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:08,778 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:08,778 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:08,778 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:08,778 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:08,778 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:08,780 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,780 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:08,780 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:08,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:08,780 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:08,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,782 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:08,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:08,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:08,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:08,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:08,785 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:08,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:08,785 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:08,786 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:08,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:08,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:57:08,789 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:08,789 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:08,789 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:08,789 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:08,789 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:08,789 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:08,833 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330f5e5c53eaf204e0a94622f3aca12b5aa to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5e5c53eaf204e0a94622f3aca12b5aa 2023-03-30 03:57:08,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:08,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148688836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148628820, max timestamp=1680148628837Num uniques 859; 2023-03-30 03:57:08,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:08,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148628820, max timestamp=1680148628837Num uniques 859; 2023-03-30 03:57:08,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:08,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148628820, max timestamp=1680148628837Num uniques 859; 2023-03-30 03:57:08,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:08,841 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/116ef7fb59854eb9a85a1ec846d98a45 2023-03-30 03:57:08,886 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:08,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:08,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:08,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:08,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:08,888 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:08,888 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:08,889 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:08,889 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:08,889 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,889 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:08,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:08,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:08,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:08,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:08,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:08,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:08,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:08,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:08,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:08,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148688888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:08,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:08,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:08,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:08,891 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:08,891 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:08,892 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:08,892 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:08,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:08,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:08,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:08,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:08,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,893 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:08,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:08,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:08,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:08,896 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:08,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a7328a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:08,896 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,896 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:08,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:08,896 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:08,897 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,897 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:08,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,897 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:08,897 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:08,897 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:08,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:08,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:08,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:08,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:08,899 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:08,899 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:08,899 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:08,899 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:08,899 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:08,900 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,900 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:08,900 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:08,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:08,900 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:08,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,902 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,902 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:08,903 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:08,902 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:08,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:08,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:08,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:08,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:08,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:08,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,903 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:08,903 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:08,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:08,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:08,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:08,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:08,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148688940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:08,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:08,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148688992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,003 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:09,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:09,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:09,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:09,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:09,006 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:09,006 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,006 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:09,006 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:09,006 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,006 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,007 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:09,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:09,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:09,008 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,008 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:09,008 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:09,008 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:09,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:09,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:09,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:09,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d619daf[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:09,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:09,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:09,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:09,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:09,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,011 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:09,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:09,013 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:09,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:09,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:09,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:09,013 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:09,014 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,014 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,014 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,015 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:09,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:09,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,016 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,017 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:09,017 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:09,017 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,017 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,117 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:09,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:09,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:09,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:09,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:09,120 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:09,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:09,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:09,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,120 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:09,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:09,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:09,122 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,122 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:09,122 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:09,122 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:09,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,123 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,123 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:09,124 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:09,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4883197b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:09,124 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:09,125 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,126 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:09,126 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:09,126 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:09,126 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:09,126 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:09,126 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:09,126 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:09,127 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,127 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,127 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,127 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,128 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,129 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:09,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:09,129 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:09,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,130 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:09,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:09,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148689144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:09,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148689194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,230 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:09,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:09,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:09,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:09,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:09,233 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:09,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:09,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:09,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,233 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,235 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,235 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,235 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,235 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:09,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:09,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:09,236 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,236 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:09,236 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:09,236 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:09,236 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:09,237 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,237 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,237 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,237 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:09,238 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:09,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71e080b6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:09,238 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,239 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:09,239 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,239 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:09,240 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:09,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:09,241 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:09,241 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:09,241 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:09,241 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:09,241 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:09,242 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,242 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,242 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,242 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,243 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,244 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,244 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,244 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,244 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,244 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,245 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:09,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:09,244 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,245 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,245 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:09,245 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:09,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:09,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/fc57890698b74efe8a47e77a63126066 2023-03-30 03:57:09,345 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:09,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:09,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:09,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:09,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:09,348 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:09,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:09,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:09,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,348 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,349 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,349 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,349 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,349 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:09,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:09,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:09,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:09,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,350 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:09,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:09,357 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:09,357 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:09,357 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:09,358 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:09,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,359 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:09,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f1dcf62[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:09,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:09,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,361 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:09,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:09,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,362 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,362 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:09,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:09,364 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:09,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:09,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:09,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:09,364 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:09,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,367 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,369 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,369 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:09,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:09,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,369 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,369 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,369 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,369 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:09,369 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,369 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:09,369 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,369 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,370 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,370 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:09,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47258 deadline: 1680148689446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,469 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:09,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:09,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:09,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:09,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:09,472 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:09,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:09,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:09,473 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,473 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,473 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,473 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,473 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,473 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:09,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:09,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:09,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:09,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:09,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:09,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:09,475 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,475 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:09,476 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:09,476 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:09,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:09,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:09,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:09,477 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,477 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:09,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e10a1cb[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:09,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:09,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,478 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:09,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:09,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:09,479 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,479 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:09,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:09,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,479 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,480 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:09,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:09,481 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:09,481 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:09,481 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:09,481 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:09,481 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:09,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,484 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,486 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:09,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:09,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:09,486 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:09,486 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:09,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47246 deadline: 1680148689498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,586 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:09,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:09,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:09,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:09,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:09,589 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:09,589 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,589 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:09,589 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:09,589 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,589 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,590 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:09,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:09,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:09,591 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,591 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:09,591 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:09,591 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:09,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,592 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:09,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e883fb3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:09,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:09,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,594 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:09,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:09,595 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,595 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:09,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:09,595 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:09,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:09,597 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:09,597 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:09,597 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:09,597 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:09,597 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:09,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:09,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:09,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:09,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:09,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:09,603 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,603 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:09,603 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:09,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:09,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:09,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/d24fd92d016d4f5c80b8078907de96f9 2023-03-30 03:57:09,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/116ef7fb59854eb9a85a1ec846d98a45 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/116ef7fb59854eb9a85a1ec846d98a45 2023-03-30 03:57:09,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/116ef7fb59854eb9a85a1ec846d98a45, entries=400, sequenceid=358, filesize=59.8 K 2023-03-30 03:57:09,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/fc57890698b74efe8a47e77a63126066 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/fc57890698b74efe8a47e77a63126066 2023-03-30 03:57:09,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/fc57890698b74efe8a47e77a63126066, entries=150, sequenceid=358, filesize=11.9 K 2023-03-30 03:57:09,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/d24fd92d016d4f5c80b8078907de96f9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d24fd92d016d4f5c80b8078907de96f9 2023-03-30 03:57:09,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d24fd92d016d4f5c80b8078907de96f9, entries=150, sequenceid=358, filesize=11.9 K 2023-03-30 03:57:09,682 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 08f755044e3e0b03b5ae847b1c94e115 in 894ms, sequenceid=358, compaction requested=true 2023-03-30 03:57:09,682 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:09,682 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:09,682 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:09,682 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:09,682 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:09,683 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:09,683 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 133060 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:09,684 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/A is initiating minor compaction (all files) 2023-03-30 03:57:09,684 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/A in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,684 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/1d41adaece1c4118a4886d81e167c9d5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/2e2610955bf0469e97954d0a62bbf20c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/5fc86cfcec4b4f6a902e1384186f426d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/116ef7fb59854eb9a85a1ec846d98a45] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=129.9 K 2023-03-30 03:57:09,684 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1d41adaece1c4118a4886d81e167c9d5, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148623743 2023-03-30 03:57:09,684 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:09,684 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/B is initiating minor compaction (all files) 2023-03-30 03:57:09,684 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 2e2610955bf0469e97954d0a62bbf20c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680148625997 2023-03-30 03:57:09,685 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/B in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,685 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/ac51d5c9fdff44368f9f5a4d297f84b6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/187b4447219e4d3f936008fa7f6fdfe2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dafc03b0ba37406b9b7307de357c206b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/fc57890698b74efe8a47e77a63126066] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=45.9 K 2023-03-30 03:57:09,685 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5fc86cfcec4b4f6a902e1384186f426d, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148626626 2023-03-30 03:57:09,685 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ac51d5c9fdff44368f9f5a4d297f84b6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148623743 2023-03-30 03:57:09,685 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 116ef7fb59854eb9a85a1ec846d98a45, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680148628755 2023-03-30 03:57:09,685 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 187b4447219e4d3f936008fa7f6fdfe2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680148625997 2023-03-30 03:57:09,686 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting dafc03b0ba37406b9b7307de357c206b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148626626 2023-03-30 03:57:09,686 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting fc57890698b74efe8a47e77a63126066, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680148628757 2023-03-30 03:57:09,699 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#B#compaction#205 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:09,703 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:09,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:09,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:09,708 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#A#compaction#204 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:09,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:09,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:09,708 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:09,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:09,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:09,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,709 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:09,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:09,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:09,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:09,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:09,713 DEBUG [Thread-1020] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x651f745a to 127.0.0.1:58033 2023-03-30 03:57:09,713 DEBUG [Thread-1020] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:09,714 DEBUG [Thread-1016] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6577ce5a to 127.0.0.1:58033 2023-03-30 03:57:09,714 DEBUG [Thread-1016] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:09,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:09,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:09,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:09,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:09,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:09,716 DEBUG [Thread-1014] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43a14bcf to 127.0.0.1:58033 2023-03-30 03:57:09,716 DEBUG [Thread-1014] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:09,718 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:09,718 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:09,718 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:09,718 DEBUG [Thread-1018] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d5bdceb to 127.0.0.1:58033 2023-03-30 03:57:09,718 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. started... 2023-03-30 03:57:09,718 DEBUG [Thread-1018] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:09,718 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-30 03:57:09,719 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:09,719 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:09,719 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:09,719 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:09,719 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:09,719 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:09,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:09,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. as already flushing 2023-03-30 03:57:09,892 DEBUG [Thread-1009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x633c81ba to 127.0.0.1:58033 2023-03-30 03:57:09,892 DEBUG [Thread-1009] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:09,918 DEBUG [Thread-1007] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5649f410 to 127.0.0.1:58033 2023-03-30 03:57:09,918 DEBUG [Thread-1005] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0df763e0 to 127.0.0.1:58033 2023-03-30 03:57:09,918 DEBUG [Thread-1005] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:09,918 DEBUG [Thread-1007] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:09,952 DEBUG [Thread-1011] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61ad5d1c to 127.0.0.1:58033 2023-03-30 03:57:09,952 DEBUG [Thread-1011] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:10,006 DEBUG [Thread-1003] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x258ba157 to 127.0.0.1:58033 2023-03-30 03:57:10,006 DEBUG [Thread-1003] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:10,130 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/e25f86ca576e4e6a8c81ab75e7e6dbb4 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/e25f86ca576e4e6a8c81ab75e7e6dbb4 2023-03-30 03:57:10,134 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/B of 08f755044e3e0b03b5ae847b1c94e115 into e25f86ca576e4e6a8c81ab75e7e6dbb4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:10,134 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:10,134 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/B, priority=12, startTime=1680148629682; duration=0sec 2023-03-30 03:57:10,134 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:10,134 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:10,135 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:10,136 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 08f755044e3e0b03b5ae847b1c94e115/C is initiating minor compaction (all files) 2023-03-30 03:57:10,136 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08f755044e3e0b03b5ae847b1c94e115/C in TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:10,136 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/075ce03895504dc8b29c456eb0378dc0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/51bbbe1bce274d1280d98d60e7d43049, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/8167d4b9221f4a708dc8ead8741c3066, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d24fd92d016d4f5c80b8078907de96f9] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp, totalSize=45.9 K 2023-03-30 03:57:10,136 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 075ce03895504dc8b29c456eb0378dc0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148623743 2023-03-30 03:57:10,137 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 51bbbe1bce274d1280d98d60e7d43049, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680148625997 2023-03-30 03:57:10,137 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8167d4b9221f4a708dc8ead8741c3066, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148626626 2023-03-30 03:57:10,137 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303308d1500caeef74b4dbe66dd2cf35a2ee2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308d1500caeef74b4dbe66dd2cf35a2ee2 2023-03-30 03:57:10,137 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d24fd92d016d4f5c80b8078907de96f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680148628757 2023-03-30 03:57:10,150 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 08f755044e3e0b03b5ae847b1c94e115#C#compaction#207 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:57:10,548 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=392, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/f2827de0a9814ae6ae3d46c4b60785f4 2023-03-30 03:57:10,558 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/a8d9ea19455e4ed3bfcf3801df90ee67 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a8d9ea19455e4ed3bfcf3801df90ee67 2023-03-30 03:57:10,563 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/C of 08f755044e3e0b03b5ae847b1c94e115 into a8d9ea19455e4ed3bfcf3801df90ee67(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:10,563 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:10,563 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/C, priority=12, startTime=1680148629683; duration=0sec 2023-03-30 03:57:10,563 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:10,959 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/7f5966ab8769460fb093f234e17a03da as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/7f5966ab8769460fb093f234e17a03da 2023-03-30 03:57:10,962 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/a9a947c37468461ab1734eac3b351cfe 2023-03-30 03:57:10,967 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08f755044e3e0b03b5ae847b1c94e115/A of 08f755044e3e0b03b5ae847b1c94e115 into 7f5966ab8769460fb093f234e17a03da(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-30 03:57:10,967 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:10,967 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115., storeName=08f755044e3e0b03b5ae847b1c94e115/A, priority=12, startTime=1680148629682; duration=1sec 2023-03-30 03:57:10,967 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:11,376 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/de1d675d3ef14088af7b906eeae2a552 2023-03-30 03:57:11,389 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/f2827de0a9814ae6ae3d46c4b60785f4 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/f2827de0a9814ae6ae3d46c4b60785f4 2023-03-30 03:57:11,395 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/f2827de0a9814ae6ae3d46c4b60785f4, entries=150, sequenceid=392, filesize=25.4 K 2023-03-30 03:57:11,396 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/a9a947c37468461ab1734eac3b351cfe as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a9a947c37468461ab1734eac3b351cfe 2023-03-30 03:57:11,401 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a9a947c37468461ab1734eac3b351cfe, entries=150, sequenceid=392, filesize=11.9 K 2023-03-30 03:57:11,402 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/de1d675d3ef14088af7b906eeae2a552 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/de1d675d3ef14088af7b906eeae2a552 2023-03-30 03:57:11,406 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/de1d675d3ef14088af7b906eeae2a552, entries=150, sequenceid=392, filesize=11.9 K 2023-03-30 03:57:11,407 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=33.54 KB/34350 for 08f755044e3e0b03b5ae847b1c94e115 in 1689ms, sequenceid=392, compaction requested=false 2023-03-30 03:57:11,407 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:11,407 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:11,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:11,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:11,408 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,408 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:11,408 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:11,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:11,409 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:11,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:11,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:11,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:11,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b98e7e4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:11,411 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:11,411 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:11,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:11,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:11,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:11,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:11,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:11,415 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:11,415 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:11,415 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:11,415 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:11,415 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:11,416 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:11,416 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:11,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:11,417 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:11,417 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:11,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:11,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:11,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:11,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:11,420 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:11,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:11,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:11,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:11,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,420 DEBUG [Thread-1013] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:11,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:11,420 DEBUG [Thread-1013] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:11,420 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:11,520 DEBUG [Thread-1013] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:11,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 141 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 5996 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 5687 2023-03-30 03:57:11,522 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 03:57:11,523 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 2419 2023-03-30 03:57:11,523 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 7257 rows 2023-03-30 03:57:11,523 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 2413 2023-03-30 03:57:11,523 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 7239 rows 2023-03-30 03:57:11,523 INFO [Listener at localhost.localdomain/37215] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 03:57:11,523 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31388307 to 127.0.0.1:58033 2023-03-30 03:57:11,523 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:11,525 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 03:57:11,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-30 03:57:11,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:11,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 03:57:11,532 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148631532"}]},"ts":"1680148631532"} 2023-03-30 03:57:11,533 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 03:57:11,534 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 03:57:11,534 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, UNASSIGN}] 2023-03-30 03:57:11,535 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, UNASSIGN 2023-03-30 03:57:11,536 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=08f755044e3e0b03b5ae847b1c94e115, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:11,536 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148631535"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148631535"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148631535"}]},"ts":"1680148631535"} 2023-03-30 03:57:11,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:57:11,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 03:57:11,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:11,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 08f755044e3e0b03b5ae847b1c94e115, disabling compactions & flushes 2023-03-30 03:57:11,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:11,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:11,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. after waiting 0 ms 2023-03-30 03:57:11,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:11,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 08f755044e3e0b03b5ae847b1c94e115 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 03:57:11,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=A 2023-03-30 03:57:11,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:11,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=B 2023-03-30 03:57:11,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:11,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08f755044e3e0b03b5ae847b1c94e115, store=C 2023-03-30 03:57:11,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:11,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 03:57:12,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330f5adc16e0e7e431ab1f49bef05e57935 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5adc16e0e7e431ab1f49bef05e57935 2023-03-30 03:57:12,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 03:57:12,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=403, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/6e63ad3ca4974eaa827ae298dffd26fd 2023-03-30 03:57:12,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 03:57:12,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/d88879c09bb94af6bafc76eec61399d1 2023-03-30 03:57:13,368 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/60d99095f93d48ae8914e6231a742bfb 2023-03-30 03:57:13,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/A/6e63ad3ca4974eaa827ae298dffd26fd as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6e63ad3ca4974eaa827ae298dffd26fd 2023-03-30 03:57:13,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6e63ad3ca4974eaa827ae298dffd26fd, entries=100, sequenceid=403, filesize=18.6 K 2023-03-30 03:57:13,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/B/d88879c09bb94af6bafc76eec61399d1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d88879c09bb94af6bafc76eec61399d1 2023-03-30 03:57:13,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d88879c09bb94af6bafc76eec61399d1, entries=100, sequenceid=403, filesize=9.5 K 2023-03-30 03:57:13,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/.tmp/C/60d99095f93d48ae8914e6231a742bfb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/60d99095f93d48ae8914e6231a742bfb 2023-03-30 03:57:13,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/60d99095f93d48ae8914e6231a742bfb, entries=100, sequenceid=403, filesize=9.5 K 2023-03-30 03:57:13,397 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 08f755044e3e0b03b5ae847b1c94e115 in 1708ms, sequenceid=403, compaction requested=true 2023-03-30 03:57:13,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/72a4e115137d493e802dbf6f036acb3e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/54b242ed2ff64545b0d438f08d897e61, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bd836302b0474427901e62e3885e4d90, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e83a79c82ce34239a0e5535c0d97299a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6d440c9507cb49fb908d4f3e24b2ed30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/01c93affa38c4cb1b950da3bd7990943, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/51808ce9c1f7472e9e133ef9954a5b30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/768a3e4c1cc94d52b92b3a676596833e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ef4f723f9cb749539602fb6c5732beed, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/9bc713a811c54ce1aa3134b762571533, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/75db632ea04e42f09ea224897f0036cf, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e49ac3e74ff7480ab39d4082cae5701a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/98639558334345a5b09ad017f45ba49f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91289384c50b4b2ca3b07ebc479fd8f0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/decc2ab455014630bbda3aa8bbd35884, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91848ccddcae425c80c3f5f3b42fc1dd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/eb56c3348f09459ab1d323ca6e420e0c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bec961f4db6047e4a6eec8d5da6393b9, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/79d4cce3eca5445899a42d06995512de, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/fe64b3e7d51a4f5b9838e85539c6f95c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ad54ccc683f940c4ba15f8e74d5ba103, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/1d41adaece1c4118a4886d81e167c9d5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/2e2610955bf0469e97954d0a62bbf20c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/5fc86cfcec4b4f6a902e1384186f426d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/116ef7fb59854eb9a85a1ec846d98a45] to archive 2023-03-30 03:57:13,421 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:57:13,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/72a4e115137d493e802dbf6f036acb3e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/72a4e115137d493e802dbf6f036acb3e 2023-03-30 03:57:13,423 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/54b242ed2ff64545b0d438f08d897e61 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/54b242ed2ff64545b0d438f08d897e61 2023-03-30 03:57:13,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bd836302b0474427901e62e3885e4d90 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bd836302b0474427901e62e3885e4d90 2023-03-30 03:57:13,425 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e83a79c82ce34239a0e5535c0d97299a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e83a79c82ce34239a0e5535c0d97299a 2023-03-30 03:57:13,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6d440c9507cb49fb908d4f3e24b2ed30 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6d440c9507cb49fb908d4f3e24b2ed30 2023-03-30 03:57:13,427 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/01c93affa38c4cb1b950da3bd7990943 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/01c93affa38c4cb1b950da3bd7990943 2023-03-30 03:57:13,428 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/51808ce9c1f7472e9e133ef9954a5b30 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/51808ce9c1f7472e9e133ef9954a5b30 2023-03-30 03:57:13,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/768a3e4c1cc94d52b92b3a676596833e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/768a3e4c1cc94d52b92b3a676596833e 2023-03-30 03:57:13,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ef4f723f9cb749539602fb6c5732beed to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ef4f723f9cb749539602fb6c5732beed 2023-03-30 03:57:13,431 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/9bc713a811c54ce1aa3134b762571533 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/9bc713a811c54ce1aa3134b762571533 2023-03-30 03:57:13,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/75db632ea04e42f09ea224897f0036cf to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/75db632ea04e42f09ea224897f0036cf 2023-03-30 03:57:13,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e49ac3e74ff7480ab39d4082cae5701a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/e49ac3e74ff7480ab39d4082cae5701a 2023-03-30 03:57:13,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/98639558334345a5b09ad017f45ba49f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/98639558334345a5b09ad017f45ba49f 2023-03-30 03:57:13,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91289384c50b4b2ca3b07ebc479fd8f0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91289384c50b4b2ca3b07ebc479fd8f0 2023-03-30 03:57:13,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/decc2ab455014630bbda3aa8bbd35884 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/decc2ab455014630bbda3aa8bbd35884 2023-03-30 03:57:13,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91848ccddcae425c80c3f5f3b42fc1dd to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/91848ccddcae425c80c3f5f3b42fc1dd 2023-03-30 03:57:13,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/eb56c3348f09459ab1d323ca6e420e0c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/eb56c3348f09459ab1d323ca6e420e0c 2023-03-30 03:57:13,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bec961f4db6047e4a6eec8d5da6393b9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/bec961f4db6047e4a6eec8d5da6393b9 2023-03-30 03:57:13,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/79d4cce3eca5445899a42d06995512de to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/79d4cce3eca5445899a42d06995512de 2023-03-30 03:57:13,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/fe64b3e7d51a4f5b9838e85539c6f95c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/fe64b3e7d51a4f5b9838e85539c6f95c 2023-03-30 03:57:13,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ad54ccc683f940c4ba15f8e74d5ba103 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/ad54ccc683f940c4ba15f8e74d5ba103 2023-03-30 03:57:13,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/1d41adaece1c4118a4886d81e167c9d5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/1d41adaece1c4118a4886d81e167c9d5 2023-03-30 03:57:13,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/2e2610955bf0469e97954d0a62bbf20c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/2e2610955bf0469e97954d0a62bbf20c 2023-03-30 03:57:13,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/5fc86cfcec4b4f6a902e1384186f426d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/5fc86cfcec4b4f6a902e1384186f426d 2023-03-30 03:57:13,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/116ef7fb59854eb9a85a1ec846d98a45 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/116ef7fb59854eb9a85a1ec846d98a45 2023-03-30 03:57:13,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/7a8871ffabb243978dabdf05a62580db, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b400ec351baf4ad887909cb5565b4d51, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/67021198b3a142729b3d724ae61842e2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4464394dbe3c4dd8b2d92120aac76220, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6abbe1929db8451f9b7e1b2cd9ebfc14, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/0d2d0d097a5745bd98a9b307100eeadb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/24670a1e6f4241aeb14a86e0c2c1bd27, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4d09419acffc4caa9878056c285b669b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/77eb8a5ff61042b3a27687f186faf56d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4f2331768da844aca7001af4d006d120, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d6ca92c04699493bbdcfa22cb560be48, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6f92798fee7a4d538c3e173fa3a3d2c4, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/bde6097894f8466aa6b83563bf545742, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/f82ae02d71124fad8c8594fe11cf8714, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b3283b5f9b824fd9b266d8f9e939f069, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a0b7ae4fa3ca464fbf07d71fc9eabd4c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5745e69e63aa4158b404579251a96a15, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5bd78040882d487dbab796f866fe2731, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/af7bd47377f148f3a7599540c47fc9ae, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dd02596f03db46b390515db8bfdfb9ca, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/ac51d5c9fdff44368f9f5a4d297f84b6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/34caa667579b450793dacc0de1f6e80e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/187b4447219e4d3f936008fa7f6fdfe2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dafc03b0ba37406b9b7307de357c206b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/fc57890698b74efe8a47e77a63126066] to archive 2023-03-30 03:57:13,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:57:13,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/7a8871ffabb243978dabdf05a62580db to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/7a8871ffabb243978dabdf05a62580db 2023-03-30 03:57:13,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b400ec351baf4ad887909cb5565b4d51 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b400ec351baf4ad887909cb5565b4d51 2023-03-30 03:57:13,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/67021198b3a142729b3d724ae61842e2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/67021198b3a142729b3d724ae61842e2 2023-03-30 03:57:13,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4464394dbe3c4dd8b2d92120aac76220 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4464394dbe3c4dd8b2d92120aac76220 2023-03-30 03:57:13,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6abbe1929db8451f9b7e1b2cd9ebfc14 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6abbe1929db8451f9b7e1b2cd9ebfc14 2023-03-30 03:57:13,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/0d2d0d097a5745bd98a9b307100eeadb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/0d2d0d097a5745bd98a9b307100eeadb 2023-03-30 03:57:13,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/24670a1e6f4241aeb14a86e0c2c1bd27 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/24670a1e6f4241aeb14a86e0c2c1bd27 2023-03-30 03:57:13,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4d09419acffc4caa9878056c285b669b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4d09419acffc4caa9878056c285b669b 2023-03-30 03:57:13,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/77eb8a5ff61042b3a27687f186faf56d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/77eb8a5ff61042b3a27687f186faf56d 2023-03-30 03:57:13,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4f2331768da844aca7001af4d006d120 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/4f2331768da844aca7001af4d006d120 2023-03-30 03:57:13,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d6ca92c04699493bbdcfa22cb560be48 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d6ca92c04699493bbdcfa22cb560be48 2023-03-30 03:57:13,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6f92798fee7a4d538c3e173fa3a3d2c4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/6f92798fee7a4d538c3e173fa3a3d2c4 2023-03-30 03:57:13,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/bde6097894f8466aa6b83563bf545742 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/bde6097894f8466aa6b83563bf545742 2023-03-30 03:57:13,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/f82ae02d71124fad8c8594fe11cf8714 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/f82ae02d71124fad8c8594fe11cf8714 2023-03-30 03:57:13,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b3283b5f9b824fd9b266d8f9e939f069 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/b3283b5f9b824fd9b266d8f9e939f069 2023-03-30 03:57:13,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a0b7ae4fa3ca464fbf07d71fc9eabd4c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a0b7ae4fa3ca464fbf07d71fc9eabd4c 2023-03-30 03:57:13,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5745e69e63aa4158b404579251a96a15 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5745e69e63aa4158b404579251a96a15 2023-03-30 03:57:13,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5bd78040882d487dbab796f866fe2731 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/5bd78040882d487dbab796f866fe2731 2023-03-30 03:57:13,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/af7bd47377f148f3a7599540c47fc9ae to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/af7bd47377f148f3a7599540c47fc9ae 2023-03-30 03:57:13,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dd02596f03db46b390515db8bfdfb9ca to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dd02596f03db46b390515db8bfdfb9ca 2023-03-30 03:57:13,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/ac51d5c9fdff44368f9f5a4d297f84b6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/ac51d5c9fdff44368f9f5a4d297f84b6 2023-03-30 03:57:13,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/34caa667579b450793dacc0de1f6e80e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/34caa667579b450793dacc0de1f6e80e 2023-03-30 03:57:13,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/187b4447219e4d3f936008fa7f6fdfe2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/187b4447219e4d3f936008fa7f6fdfe2 2023-03-30 03:57:13,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dafc03b0ba37406b9b7307de357c206b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/dafc03b0ba37406b9b7307de357c206b 2023-03-30 03:57:13,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/fc57890698b74efe8a47e77a63126066 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/fc57890698b74efe8a47e77a63126066 2023-03-30 03:57:13,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c367223ee0f44a892210f88c3edd9db, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/04ecc2b5286a48779eff5395e353271c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/885b3a0321a544d495ebc2df7725698a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df1e5b7ac6c8449c89660a2a655da07b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d376edf12b614a7baab887eac477740e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/541317265be04f31a766dbbe2c42238a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/134166b58b72430fb78566ab3682c204, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b3429102086b4beeaa45ee72a3b985eb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/e91773e578a44a0e811ea0f54b57f0b6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c9964df93b74471a0de8200115b8514, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/5eaa1055fb2a423993a2aa59eaa7872d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/fc1eb5e4d87a43338f7549dd35555ee4, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/525d6ff9427e4790afb16b7294e65a0b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/2c0fcc8395db460bb3a45b9aaea1b8d2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a26b065a99bc44b3aeb893c6bcb5f699, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d2cd9ba882d840d08346a5ae48c72559, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a0c4a521f872469382b26ed937b7c133, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b945a87ea6a841368c55f62cc55b311e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/27c5ae665fdb49baa75553c845d782c5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df5a79c49abb443ab061988b8194c0e1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/075ce03895504dc8b29c456eb0378dc0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/db0219d577f84f22b20d5c63d0b042f5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/51bbbe1bce274d1280d98d60e7d43049, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/8167d4b9221f4a708dc8ead8741c3066, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d24fd92d016d4f5c80b8078907de96f9] to archive 2023-03-30 03:57:13,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:57:13,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c367223ee0f44a892210f88c3edd9db to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c367223ee0f44a892210f88c3edd9db 2023-03-30 03:57:13,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/04ecc2b5286a48779eff5395e353271c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/04ecc2b5286a48779eff5395e353271c 2023-03-30 03:57:13,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/885b3a0321a544d495ebc2df7725698a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/885b3a0321a544d495ebc2df7725698a 2023-03-30 03:57:13,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df1e5b7ac6c8449c89660a2a655da07b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df1e5b7ac6c8449c89660a2a655da07b 2023-03-30 03:57:13,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d376edf12b614a7baab887eac477740e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d376edf12b614a7baab887eac477740e 2023-03-30 03:57:13,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/541317265be04f31a766dbbe2c42238a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/541317265be04f31a766dbbe2c42238a 2023-03-30 03:57:13,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/134166b58b72430fb78566ab3682c204 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/134166b58b72430fb78566ab3682c204 2023-03-30 03:57:13,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b3429102086b4beeaa45ee72a3b985eb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b3429102086b4beeaa45ee72a3b985eb 2023-03-30 03:57:13,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/e91773e578a44a0e811ea0f54b57f0b6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/e91773e578a44a0e811ea0f54b57f0b6 2023-03-30 03:57:13,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c9964df93b74471a0de8200115b8514 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/9c9964df93b74471a0de8200115b8514 2023-03-30 03:57:13,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/5eaa1055fb2a423993a2aa59eaa7872d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/5eaa1055fb2a423993a2aa59eaa7872d 2023-03-30 03:57:13,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/fc1eb5e4d87a43338f7549dd35555ee4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/fc1eb5e4d87a43338f7549dd35555ee4 2023-03-30 03:57:13,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/525d6ff9427e4790afb16b7294e65a0b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/525d6ff9427e4790afb16b7294e65a0b 2023-03-30 03:57:13,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/2c0fcc8395db460bb3a45b9aaea1b8d2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/2c0fcc8395db460bb3a45b9aaea1b8d2 2023-03-30 03:57:13,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a26b065a99bc44b3aeb893c6bcb5f699 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a26b065a99bc44b3aeb893c6bcb5f699 2023-03-30 03:57:13,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d2cd9ba882d840d08346a5ae48c72559 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d2cd9ba882d840d08346a5ae48c72559 2023-03-30 03:57:13,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a0c4a521f872469382b26ed937b7c133 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a0c4a521f872469382b26ed937b7c133 2023-03-30 03:57:13,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b945a87ea6a841368c55f62cc55b311e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/b945a87ea6a841368c55f62cc55b311e 2023-03-30 03:57:13,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/27c5ae665fdb49baa75553c845d782c5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/27c5ae665fdb49baa75553c845d782c5 2023-03-30 03:57:13,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df5a79c49abb443ab061988b8194c0e1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/df5a79c49abb443ab061988b8194c0e1 2023-03-30 03:57:13,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/075ce03895504dc8b29c456eb0378dc0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/075ce03895504dc8b29c456eb0378dc0 2023-03-30 03:57:13,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/db0219d577f84f22b20d5c63d0b042f5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/db0219d577f84f22b20d5c63d0b042f5 2023-03-30 03:57:13,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/51bbbe1bce274d1280d98d60e7d43049 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/51bbbe1bce274d1280d98d60e7d43049 2023-03-30 03:57:13,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/8167d4b9221f4a708dc8ead8741c3066 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/8167d4b9221f4a708dc8ead8741c3066 2023-03-30 03:57:13,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d24fd92d016d4f5c80b8078907de96f9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/d24fd92d016d4f5c80b8078907de96f9 2023-03-30 03:57:13,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/recovered.edits/406.seqid, newMaxSeqId=406, maxSeqId=4 2023-03-30 03:57:13,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115. 2023-03-30 03:57:13,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 08f755044e3e0b03b5ae847b1c94e115: 2023-03-30 03:57:13,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:13,502 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=08f755044e3e0b03b5ae847b1c94e115, regionState=CLOSED 2023-03-30 03:57:13,502 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148633502"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148633502"}]},"ts":"1680148633502"} 2023-03-30 03:57:13,504 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-30 03:57:13,504 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 08f755044e3e0b03b5ae847b1c94e115, server=jenkins-hbase20.apache.org,41157,1680148568851 in 1.9670 sec 2023-03-30 03:57:13,506 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-30 03:57:13,506 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08f755044e3e0b03b5ae847b1c94e115, UNASSIGN in 1.9700 sec 2023-03-30 03:57:13,506 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148633506"}]},"ts":"1680148633506"} 2023-03-30 03:57:13,507 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 03:57:13,508 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 03:57:13,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9830 sec 2023-03-30 03:57:13,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 03:57:13,641 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-30 03:57:13,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-30 03:57:13,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:13,644 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:13,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-30 03:57:13,645 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:13,649 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:13,651 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/recovered.edits] 2023-03-30 03:57:13,654 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6e63ad3ca4974eaa827ae298dffd26fd to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/6e63ad3ca4974eaa827ae298dffd26fd 2023-03-30 03:57:13,656 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/7f5966ab8769460fb093f234e17a03da to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/7f5966ab8769460fb093f234e17a03da 2023-03-30 03:57:13,657 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/f2827de0a9814ae6ae3d46c4b60785f4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/A/f2827de0a9814ae6ae3d46c4b60785f4 2023-03-30 03:57:13,659 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a9a947c37468461ab1734eac3b351cfe to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/a9a947c37468461ab1734eac3b351cfe 2023-03-30 03:57:13,660 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d88879c09bb94af6bafc76eec61399d1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/d88879c09bb94af6bafc76eec61399d1 2023-03-30 03:57:13,661 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/e25f86ca576e4e6a8c81ab75e7e6dbb4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/B/e25f86ca576e4e6a8c81ab75e7e6dbb4 2023-03-30 03:57:13,662 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/60d99095f93d48ae8914e6231a742bfb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/60d99095f93d48ae8914e6231a742bfb 2023-03-30 03:57:13,663 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a8d9ea19455e4ed3bfcf3801df90ee67 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/a8d9ea19455e4ed3bfcf3801df90ee67 2023-03-30 03:57:13,664 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/de1d675d3ef14088af7b906eeae2a552 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/C/de1d675d3ef14088af7b906eeae2a552 2023-03-30 03:57:13,666 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/recovered.edits/406.seqid to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115/recovered.edits/406.seqid 2023-03-30 03:57:13,667 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/08f755044e3e0b03b5ae847b1c94e115 2023-03-30 03:57:13,667 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:57:13,667 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 03:57:13,668 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-30 03:57:13,671 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303300e29bec612ed4e64b0b06b8b2d7748ad to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303300e29bec612ed4e64b0b06b8b2d7748ad 2023-03-30 03:57:13,672 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303143310f7bad44e1afe9b3e8cb90102c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303143310f7bad44e1afe9b3e8cb90102c 2023-03-30 03:57:13,673 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303363a8b3e880426295137909069f82a0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303363a8b3e880426295137909069f82a0 2023-03-30 03:57:13,673 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033038d7b12a18894dac92d38b6777cd3c9b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033038d7b12a18894dac92d38b6777cd3c9b 2023-03-30 03:57:13,674 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033050abb6cf0f2f4682a13d8b242f4c09f1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033050abb6cf0f2f4682a13d8b242f4c09f1 2023-03-30 03:57:13,675 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330517ee75931d94b949e62e63d31e9cf5d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330517ee75931d94b949e62e63d31e9cf5d 2023-03-30 03:57:13,676 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033051b6ee94af7443d794bcaeaecf625998 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033051b6ee94af7443d794bcaeaecf625998 2023-03-30 03:57:13,676 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033052690c91ab6349fca4e7832a2c214a53 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033052690c91ab6349fca4e7832a2c214a53 2023-03-30 03:57:13,677 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033067c1aaafa508421f8f785b3c4f17c381 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033067c1aaafa508421f8f785b3c4f17c381 2023-03-30 03:57:13,678 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308d1500caeef74b4dbe66dd2cf35a2ee2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308d1500caeef74b4dbe66dd2cf35a2ee2 2023-03-30 03:57:13,679 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033094f86564ac3d4ec6a37750cf012c2c9d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033094f86564ac3d4ec6a37750cf012c2c9d 2023-03-30 03:57:13,680 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330abdb842b08d04fc18e3ee1236bf53b6b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330abdb842b08d04fc18e3ee1236bf53b6b 2023-03-30 03:57:13,681 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c505ef666a4d4cbc9617103dee0180eb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c505ef666a4d4cbc9617103dee0180eb 2023-03-30 03:57:13,682 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d19d91ce11e1499999da90fe9ae4f365 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d19d91ce11e1499999da90fe9ae4f365 2023-03-30 03:57:13,684 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d4e332f3f86140b98fea8f8c2f2d6af4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d4e332f3f86140b98fea8f8c2f2d6af4 2023-03-30 03:57:13,685 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d58a605de58f4381b3ef1540f7c6a911 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d58a605de58f4381b3ef1540f7c6a911 2023-03-30 03:57:13,686 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dea487304a7944c6b146f26b32971d64 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dea487304a7944c6b146f26b32971d64 2023-03-30 03:57:13,688 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330e67e1e987a2d460581987c006ef8f056 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330e67e1e987a2d460581987c006ef8f056 2023-03-30 03:57:13,689 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5adc16e0e7e431ab1f49bef05e57935 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5adc16e0e7e431ab1f49bef05e57935 2023-03-30 03:57:13,690 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5e5c53eaf204e0a94622f3aca12b5aa to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5e5c53eaf204e0a94622f3aca12b5aa 2023-03-30 03:57:13,692 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330fb626ca6791f4c6595564c789cf01d34 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330fb626ca6791f4c6595564c789cf01d34 2023-03-30 03:57:13,692 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 03:57:13,694 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:13,696 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 03:57:13,698 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 03:57:13,699 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:13,699 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 03:57:13,699 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680148633699"}]},"ts":"9223372036854775807"} 2023-03-30 03:57:13,701 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 03:57:13,701 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 08f755044e3e0b03b5ae847b1c94e115, NAME => 'TestAcidGuarantees,,1680148606528.08f755044e3e0b03b5ae847b1c94e115.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 03:57:13,701 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 03:57:13,702 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680148633702"}]},"ts":"9223372036854775807"} 2023-03-30 03:57:13,703 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 03:57:13,705 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:13,705 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 63 msec 2023-03-30 03:57:13,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-30 03:57:13,746 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-30 03:57:13,763 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=228 (was 230), OpenFileDescriptor=561 (was 536) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=355 (was 325) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=10224 (was 10404) 2023-03-30 03:57:13,777 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228, OpenFileDescriptor=561, MaxFileDescriptor=60000, SystemLoadAverage=355, ProcessCount=166, AvailableMemoryMB=10224 2023-03-30 03:57:13,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:57:13,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:57:13,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:13,782 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 03:57:13,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-30 03:57:13,783 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 03:57:13,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 03:57:13,784 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:13,784 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443 empty. 2023-03-30 03:57:13,784 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:13,784 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:57:13,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 03:57:14,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 03:57:14,195 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:57:14,202 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b92c602cdf3a8ada8ca7fd94e7d5b443, NAME => 'TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp 2023-03-30 03:57:14,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 03:57:14,611 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:57:14,611 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b92c602cdf3a8ada8ca7fd94e7d5b443, disabling compactions & flushes 2023-03-30 03:57:14,611 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:14,611 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:14,611 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. after waiting 0 ms 2023-03-30 03:57:14,611 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:14,611 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:14,611 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:14,614 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 03:57:14,615 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148634614"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148634614"}]},"ts":"1680148634614"} 2023-03-30 03:57:14,615 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 03:57:14,616 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 03:57:14,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148634616"}]},"ts":"1680148634616"} 2023-03-30 03:57:14,617 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 03:57:14,619 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b92c602cdf3a8ada8ca7fd94e7d5b443, ASSIGN}] 2023-03-30 03:57:14,620 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b92c602cdf3a8ada8ca7fd94e7d5b443, ASSIGN 2023-03-30 03:57:14,621 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b92c602cdf3a8ada8ca7fd94e7d5b443, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=false 2023-03-30 03:57:14,772 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=b92c602cdf3a8ada8ca7fd94e7d5b443, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:14,772 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148634772"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148634772"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148634772"}]},"ts":"1680148634772"} 2023-03-30 03:57:14,774 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:57:14,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 03:57:14,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:14,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b92c602cdf3a8ada8ca7fd94e7d5b443, NAME => 'TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:57:14,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:57:14,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,933 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,936 DEBUG [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A 2023-03-30 03:57:14,936 DEBUG [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A 2023-03-30 03:57:14,937 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:14,937 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b92c602cdf3a8ada8ca7fd94e7d5b443 columnFamilyName A 2023-03-30 03:57:14,938 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.HStore(310): Store=b92c602cdf3a8ada8ca7fd94e7d5b443/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:14,938 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,940 DEBUG [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B 2023-03-30 03:57:14,940 DEBUG [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B 2023-03-30 03:57:14,940 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:14,940 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b92c602cdf3a8ada8ca7fd94e7d5b443 columnFamilyName B 2023-03-30 03:57:14,941 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.HStore(310): Store=b92c602cdf3a8ada8ca7fd94e7d5b443/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:14,941 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,942 DEBUG [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C 2023-03-30 03:57:14,942 DEBUG [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C 2023-03-30 03:57:14,942 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:14,942 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b92c602cdf3a8ada8ca7fd94e7d5b443 columnFamilyName C 2023-03-30 03:57:14,943 INFO [StoreOpener-b92c602cdf3a8ada8ca7fd94e7d5b443-1] regionserver.HStore(310): Store=b92c602cdf3a8ada8ca7fd94e7d5b443/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:14,943 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:14,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:57:14,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:14,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:57:14,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened b92c602cdf3a8ada8ca7fd94e7d5b443; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72643291, jitterRate=0.0824693888425827}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:57:14,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:14,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., pid=30, masterSystemTime=1680148634925 2023-03-30 03:57:14,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:14,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:14,952 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=b92c602cdf3a8ada8ca7fd94e7d5b443, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:14,952 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148634952"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148634952"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148634952"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148634952"}]},"ts":"1680148634952"} 2023-03-30 03:57:14,955 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-30 03:57:14,955 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 in 180 msec 2023-03-30 03:57:14,957 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-30 03:57:14,957 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b92c602cdf3a8ada8ca7fd94e7d5b443, ASSIGN in 336 msec 2023-03-30 03:57:14,957 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 03:57:14,958 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148634957"}]},"ts":"1680148634957"} 2023-03-30 03:57:14,959 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 03:57:14,961 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 03:57:14,962 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1820 sec 2023-03-30 03:57:15,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 03:57:15,891 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-30 03:57:15,892 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x136e067d to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,901 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d3f4b11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,903 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:15,905 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40642, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:15,907 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 03:57:15,910 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48414, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 03:57:15,912 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x12394d27 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,918 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7187fcf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,918 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x29a548ac to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,925 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@712d78ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,925 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x43184c57 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,933 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@407b9cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,933 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x7e3efec6 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,941 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f456fa2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,942 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x6b7f716c to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,950 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62be7a0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,950 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x7abacd49 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,959 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b458e37, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,959 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x57d25552 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,966 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bae1a64, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,967 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x13695730 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,975 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5654f8ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,975 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x0993cc7d to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,983 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@509adaa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,984 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x69b9b3fd to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:15,992 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e5e7a93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:15,998 DEBUG [hconnection-0x734e7d63-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:15,998 DEBUG [hconnection-0x40be8aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:15,998 DEBUG [hconnection-0x7a706b09-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:15,999 DEBUG [hconnection-0x781b8d1a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:16,001 DEBUG [hconnection-0x119bc58a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:16,001 DEBUG [hconnection-0x6dc6a494-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:16,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:16,001 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40666, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,001 DEBUG [hconnection-0x38368c32-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:16,002 DEBUG [hconnection-0x1ad5bd98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:16,002 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40682, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,010 DEBUG [hconnection-0x519b6d31-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:16,010 DEBUG [hconnection-0x75dddee7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:16,011 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40658, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,011 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40648, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,012 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40672, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,012 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40670, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,012 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40700, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,012 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40714, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,012 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40686, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,017 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40710, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:16,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:16,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:16,017 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:16,017 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:16,017 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:16,017 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:16,019 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,019 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:16,022 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:16,022 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:16,023 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,023 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:16,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:16,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:16,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:16,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:16,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:16,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:16,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:16,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:16,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:16,024 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:16,024 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:16,027 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-30 03:57:16,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:16,028 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:16,028 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:16,028 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:16,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:16,028 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:16,028 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:16,028 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:16,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148696043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148696046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148636035, max timestamp=1680148636045Num uniques 859; 2023-03-30 03:57:16,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:16,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148636035, max timestamp=1680148636045Num uniques 859; 2023-03-30 03:57:16,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:16,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148636035, max timestamp=1680148636045Num uniques 859; 2023-03-30 03:57:16,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:16,064 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/45945dadeae7402ab3d748628711a40e 2023-03-30 03:57:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680148636062, max timestamp=1680148636074Num uniques 1018; 2023-03-30 03:57:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-30 03:57:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680148636062, max timestamp=1680148636074Num uniques 1018; 2023-03-30 03:57:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-30 03:57:16,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680148636062, max timestamp=1680148636074Num uniques 1018; 2023-03-30 03:57:16,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 03:57:16,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148696097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148696097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148696098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148696151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148696153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148696199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148696200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148696200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148696354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148696355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148696401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148696402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148696402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,493 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/8976e58c66964e96bb7590e95ca88836 2023-03-30 03:57:16,505 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/db3c78b0ec60449399d36497487c7399 2023-03-30 03:57:16,508 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/45945dadeae7402ab3d748628711a40e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/45945dadeae7402ab3d748628711a40e 2023-03-30 03:57:16,511 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/45945dadeae7402ab3d748628711a40e, entries=150, sequenceid=11, filesize=11.6 K 2023-03-30 03:57:16,512 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/8976e58c66964e96bb7590e95ca88836 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8976e58c66964e96bb7590e95ca88836 2023-03-30 03:57:16,516 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8976e58c66964e96bb7590e95ca88836, entries=150, sequenceid=11, filesize=11.6 K 2023-03-30 03:57:16,516 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/db3c78b0ec60449399d36497487c7399 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/db3c78b0ec60449399d36497487c7399 2023-03-30 03:57:16,519 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/db3c78b0ec60449399d36497487c7399, entries=150, sequenceid=11, filesize=11.6 K 2023-03-30 03:57:16,520 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=143.13 KB/146560 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 493ms, sequenceid=11, compaction requested=false 2023-03-30 03:57:16,520 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:16,520 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:16,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:16,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:16,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:16,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:16,521 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:16,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:16,522 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:16,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:16,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a2f4d2a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:16,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:16,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:16,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:16,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:16,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:16,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:16,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:16,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:16,524 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:16,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:16,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:16,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:16,527 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:16,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:16,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:16,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:16,527 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:16,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:16,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:16,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:16,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:16,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:16,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,529 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,531 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:16,531 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,531 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:16,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:16,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:16,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:16,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:16,531 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:16,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:16,531 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:16,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:16,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:16,531 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:16,531 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:16,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,532 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:16,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:16,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:16,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,632 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:16,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:16,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:16,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:16,635 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:16,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:16,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:16,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:16,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:16,636 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:16,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:16,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:16,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:16,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:16,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:16,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:16,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:16,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:16,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:16,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:16,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:16,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:16,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:16,637 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:16,637 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:16,637 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=143.13 KB heapSize=404.45 KB 2023-03-30 03:57:16,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:16,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:16,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:16,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:16,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:16,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:16,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:16,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:16,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148696667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148696668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148696704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148696706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148696707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148696789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148696789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148696991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:16,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:16,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148696992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,044 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/17d5be5bc5cf461087c43f19f77dbb59 2023-03-30 03:57:17,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:17,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148697210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148697211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148697212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:17,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148697295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:17,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148697295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,458 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/c4e2338479a4454da08a7034471e7046 2023-03-30 03:57:17,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:17,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148697798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:17,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148697801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,867 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/5454f0298f1a47e6ad2793b805ec3105 2023-03-30 03:57:17,871 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/17d5be5bc5cf461087c43f19f77dbb59 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17d5be5bc5cf461087c43f19f77dbb59 2023-03-30 03:57:17,877 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17d5be5bc5cf461087c43f19f77dbb59, entries=150, sequenceid=65, filesize=11.6 K 2023-03-30 03:57:17,878 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/c4e2338479a4454da08a7034471e7046 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c4e2338479a4454da08a7034471e7046 2023-03-30 03:57:17,882 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c4e2338479a4454da08a7034471e7046, entries=150, sequenceid=65, filesize=11.6 K 2023-03-30 03:57:17,882 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/5454f0298f1a47e6ad2793b805ec3105 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/5454f0298f1a47e6ad2793b805ec3105 2023-03-30 03:57:17,885 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/5454f0298f1a47e6ad2793b805ec3105, entries=150, sequenceid=65, filesize=11.6 K 2023-03-30 03:57:17,886 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~404.41 KB/414112, currentSize=40.25 KB/41220 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 1249ms, sequenceid=65, compaction requested=false 2023-03-30 03:57:17,886 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:17,886 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:17,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:17,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:17,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:17,886 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:17,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,887 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:17,887 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:17,888 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:17,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:17,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@710aa35d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:17,889 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:17,889 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:17,889 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:17,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:17,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:17,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:17,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:17,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:17,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:17,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:17,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:17,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:17,892 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:17,892 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:17,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:17,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:17,893 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:17,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:17,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:17,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:17,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:17,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:17,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:17,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,896 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:17,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:17,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:17,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:17,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:17,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:17,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:17,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:17,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:17,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:17,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:17,897 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:17,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:17,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:17,897 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:17,897 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:17,997 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:17,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:18,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:18,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:18,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:18,010 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:18,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:18,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:18,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:18,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:18,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:18,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:18,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:18,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:18,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:18,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:18,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:18,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:18,013 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:18,013 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:18,013 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:18,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:18,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:18,022 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:18,022 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:18,022 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-30 03:57:18,022 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:18,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:18,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:18,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:18,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:18,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:18,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:18,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:18,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148638227, max timestamp=1680148638233Num uniques 859; 2023-03-30 03:57:18,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148638227, max timestamp=1680148638233Num uniques 859; 2023-03-30 03:57:18,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148638227, max timestamp=1680148638233Num uniques 859; 2023-03-30 03:57:18,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:18,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:18,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:18,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148698253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148698254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680148638248, max timestamp=1680148638255Num uniques 1018; 2023-03-30 03:57:18,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 03:57:18,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680148638248, max timestamp=1680148638255Num uniques 1018; 2023-03-30 03:57:18,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 03:57:18,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680148638248, max timestamp=1680148638255Num uniques 1018; 2023-03-30 03:57:18,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 03:57:18,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148698299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148698357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148698357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148698401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,464 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/2cc8e79a9fe8434d8db4e485752b4e4e 2023-03-30 03:57:18,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148698559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148698560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148698603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148698805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148698811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148698863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148698863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:18,880 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/852f9008b2164e9db43529c43f0230a6 2023-03-30 03:57:18,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:18,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148698907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,037 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 03:57:19,288 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/d93e4e8233714e39ba1065af53cca9bd 2023-03-30 03:57:19,292 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/2cc8e79a9fe8434d8db4e485752b4e4e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2cc8e79a9fe8434d8db4e485752b4e4e 2023-03-30 03:57:19,295 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2cc8e79a9fe8434d8db4e485752b4e4e, entries=100, sequenceid=74, filesize=9.3 K 2023-03-30 03:57:19,295 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/852f9008b2164e9db43529c43f0230a6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/852f9008b2164e9db43529c43f0230a6 2023-03-30 03:57:19,298 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/852f9008b2164e9db43529c43f0230a6, entries=100, sequenceid=74, filesize=9.3 K 2023-03-30 03:57:19,299 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/d93e4e8233714e39ba1065af53cca9bd as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/d93e4e8233714e39ba1065af53cca9bd 2023-03-30 03:57:19,302 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/d93e4e8233714e39ba1065af53cca9bd, entries=100, sequenceid=74, filesize=9.3 K 2023-03-30 03:57:19,303 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=138.65 KB/141980 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 1281ms, sequenceid=74, compaction requested=true 2023-03-30 03:57:19,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:19,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:19,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:19,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:19,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:19,304 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:19,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e507f68[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:19,305 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:19,305 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,306 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:19,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:19,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:19,306 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,306 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:19,307 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:19,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:19,309 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:19,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:19,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:19,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:19,309 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:19,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,310 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,311 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,312 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:19,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:19,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,313 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:19,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:19,313 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:19,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:19,368 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=145.36 KB heapSize=414.92 KB 2023-03-30 03:57:19,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:19,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:57:19,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:19,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:57:19,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:19,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:57:19,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148699376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148699377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148699412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,413 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:19,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:19,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:19,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:19,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:19,416 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:19,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:19,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:19,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,417 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,417 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,417 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,417 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:19,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:19,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:19,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:19,418 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,418 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:19,418 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:19,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:19,419 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,420 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,420 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:19,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:19,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab1481[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:19,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,422 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,422 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:19,422 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,422 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:19,422 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:19,422 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:19,423 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:19,423 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,423 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,423 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:19,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:19,425 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:19,425 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:19,425 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:19,425 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:19,425 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:19,426 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,426 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,426 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,427 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,427 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,429 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,429 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,429 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,429 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,429 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:19,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:19,430 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,430 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,429 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,430 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,430 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,430 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,430 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:19,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:19,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148699480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148699480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,530 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:19,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:19,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:19,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:19,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:19,534 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:19,534 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,534 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:19,534 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:19,534 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,534 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,535 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,535 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,535 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:19,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:19,536 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,536 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:19,536 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:19,536 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:19,536 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:19,537 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,537 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,538 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,538 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:19,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32dd4fe3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:19,538 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:19,539 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,539 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,539 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:19,539 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,539 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:19,539 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:19,539 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:19,540 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,540 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:19,540 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,540 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:19,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:19,541 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:19,541 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:19,541 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:19,541 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:19,542 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:19,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,543 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,544 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:19,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,545 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,546 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,546 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:19,546 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,546 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,546 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,646 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:19,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:19,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:19,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:19,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:19,653 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:19,653 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,653 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:19,653 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:19,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,656 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,657 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,657 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:19,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:19,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:19,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:19,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:19,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:19,658 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:19,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:19,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:19,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:19,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:19,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:19,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:19,667 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,667 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:19,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e912ae[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:19,669 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:19,669 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,670 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,670 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:19,670 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:19,670 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:19,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:19,671 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:19,671 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,671 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:19,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:19,674 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:19,674 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:19,674 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:19,674 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:19,674 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:19,675 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,675 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,675 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,676 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:19,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:19,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,679 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,680 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:19,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:19,680 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:19,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148699683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148699684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,776 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/102a6174871c48b49086924af76af8bc 2023-03-30 03:57:19,780 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:19,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:19,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:19,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:19,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:19,783 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:19,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:19,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:19,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,784 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:19,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:19,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:19,785 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:19,785 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,785 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:19,785 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:19,785 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:19,785 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,785 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:19,785 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:19,786 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,786 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:19,786 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:19,786 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,786 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:19,786 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:19,787 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,787 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,787 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,787 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:19,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9f8424[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:19,788 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:19,788 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,789 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:19,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:19,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:19,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:19,790 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,790 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:19,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:19,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:19,792 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:19,792 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:19,793 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:19,793 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:19,793 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:19,793 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,793 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,793 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,794 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,795 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:19,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:19,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,797 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:19,797 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:19,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:19,897 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:19,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:19,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:19,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:19,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:19,901 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:19,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:19,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:19,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,902 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,902 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,902 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,902 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:19,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:19,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:19,903 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:19,903 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:19,903 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:19,903 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:19,903 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:19,903 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,903 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:19,903 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:19,903 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:19,903 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:19,904 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:19,904 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,904 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:19,904 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:19,904 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,904 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,905 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,905 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:19,906 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:19,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11ca8eca[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:19,906 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,907 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:19,907 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:19,907 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:19,907 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,907 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:19,908 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,908 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,908 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,908 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:19,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:19,910 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:19,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:19,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:19,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:19,910 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:19,911 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:19,911 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,911 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:19,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,911 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,913 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:19,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:19,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:19,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:19,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:19,915 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:19,915 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:19,915 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:19,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:19,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148699987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:19,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:19,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148699988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,015 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:20,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:20,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:20,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:20,018 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:20,018 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,018 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:20,018 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:20,019 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,019 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,019 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,019 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,020 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,020 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:20,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:20,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:20,020 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:20,020 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:20,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:20,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:20,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:20,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:20,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:20,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:20,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:20,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:20,022 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:20,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49b06947[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:20,023 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:20,023 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,024 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:20,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:20,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:20,024 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,024 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:20,025 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,025 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:20,025 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,025 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:20,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:20,026 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:20,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:20,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:20,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:20,026 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:20,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,028 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,031 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,031 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,031 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:20,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:20,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:20,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,031 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:20,031 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:20,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,031 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,131 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:20,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:20,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:20,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:20,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:20,135 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:20,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:20,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:20,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,136 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:20,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:20,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:20,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:20,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,138 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:20,142 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:20,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,142 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:20,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:20,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:20,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:20,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:20,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:20,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:20,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,149 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,149 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,149 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:20,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17927ebd[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:20,151 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:20,151 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:20,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:20,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:20,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:20,158 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:20,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:20,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:20,161 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:20,161 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:20,161 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:20,161 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:20,161 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:20,162 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,162 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,162 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,163 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,164 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:20,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,167 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:20,167 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:20,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/9b80e95288aa44819f436869e7fe3a46 2023-03-30 03:57:20,267 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:20,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:20,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:20,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:20,274 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:20,274 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,274 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:20,274 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:20,275 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,275 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,275 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:20,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:20,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:20,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:20,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:20,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,294 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:20,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:20,309 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:20,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:20,312 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:20,313 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,313 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:20,313 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:20,313 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,313 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:20,313 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:20,315 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,315 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,315 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,315 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:20,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@201b66cf[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:20,317 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:20,317 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,318 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:20,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:20,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:20,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:20,319 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:20,319 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,319 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:20,319 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:20,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:20,322 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:20,322 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:20,322 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:20,322 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:20,322 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:20,324 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,324 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,324 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,326 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,328 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:20,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:20,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,329 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,329 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,329 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,329 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:20,329 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:20,329 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:20,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148700421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,429 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:20,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:20,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:20,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:20,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:20,444 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:20,444 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,444 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:20,444 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:20,447 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,447 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,447 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,448 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,448 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:20,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:20,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:20,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:20,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:20,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:20,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:20,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:20,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:20,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:20,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:20,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,468 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:20,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656200d8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:20,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:20,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,471 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:20,471 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,471 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:20,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:20,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:20,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:20,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,472 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:20,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:20,474 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:20,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:20,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:20,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:20,474 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:20,475 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,475 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,475 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:20,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,480 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:20,480 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:20,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148700492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:20,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148700500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,581 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:20,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:20,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:20,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:20,599 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:20,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:20,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:20,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,600 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:20,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:20,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:20,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,653 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/74dc8dc94c4e40cd829460de90860860 2023-03-30 03:57:20,653 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:20,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/102a6174871c48b49086924af76af8bc as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/102a6174871c48b49086924af76af8bc 2023-03-30 03:57:20,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/102a6174871c48b49086924af76af8bc, entries=200, sequenceid=129, filesize=13.9 K 2023-03-30 03:57:20,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:20,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:20,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/9b80e95288aa44819f436869e7fe3a46 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9b80e95288aa44819f436869e7fe3a46 2023-03-30 03:57:20,664 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,665 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:20,665 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:20,665 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,665 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:20,665 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:20,665 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,665 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:20,665 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:20,666 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:20,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:20,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f52b762[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:20,669 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:20,669 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:20,671 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9b80e95288aa44819f436869e7fe3a46, entries=150, sequenceid=129, filesize=11.7 K 2023-03-30 03:57:20,673 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:20,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:20,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:20,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/74dc8dc94c4e40cd829460de90860860 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/74dc8dc94c4e40cd829460de90860860 2023-03-30 03:57:20,675 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:20,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:20,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:20,683 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:20,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:20,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:20,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:20,683 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:20,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/74dc8dc94c4e40cd829460de90860860, entries=150, sequenceid=129, filesize=11.7 K 2023-03-30 03:57:20,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~145.36 KB/148850, heapSize ~414.88 KB/424832, currentSize=33.54 KB/34350 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 1316ms, sequenceid=129, compaction requested=true 2023-03-30 03:57:20,684 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:20,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:20,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:20,684 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:20,685 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:20,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-30 03:57:20,687 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47590 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:20,687 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/A is initiating minor compaction (all files) 2023-03-30 03:57:20,687 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/A in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,687 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/45945dadeae7402ab3d748628711a40e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17d5be5bc5cf461087c43f19f77dbb59, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2cc8e79a9fe8434d8db4e485752b4e4e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/102a6174871c48b49086924af76af8bc] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=46.5 K 2023-03-30 03:57:20,688 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 45945dadeae7402ab3d748628711a40e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680148636023 2023-03-30 03:57:20,689 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 17d5be5bc5cf461087c43f19f77dbb59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680148636093 2023-03-30 03:57:20,689 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45250 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:20,689 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/B is initiating minor compaction (all files) 2023-03-30 03:57:20,689 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/B in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,690 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8976e58c66964e96bb7590e95ca88836, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c4e2338479a4454da08a7034471e7046, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/852f9008b2164e9db43529c43f0230a6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9b80e95288aa44819f436869e7fe3a46] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=44.2 K 2023-03-30 03:57:20,690 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2cc8e79a9fe8434d8db4e485752b4e4e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680148636663 2023-03-30 03:57:20,693 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 8976e58c66964e96bb7590e95ca88836, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680148636023 2023-03-30 03:57:20,693 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 102a6174871c48b49086924af76af8bc, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680148638290 2023-03-30 03:57:20,694 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting c4e2338479a4454da08a7034471e7046, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680148636093 2023-03-30 03:57:20,694 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 852f9008b2164e9db43529c43f0230a6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680148636663 2023-03-30 03:57:20,695 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9b80e95288aa44819f436869e7fe3a46, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680148638290 2023-03-30 03:57:20,698 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,698 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,698 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,702 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,705 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,705 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,705 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,705 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,705 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,706 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#A#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:20,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:20,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:20,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,706 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:20,706 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:20,717 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:20,741 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/fd60a5ebb7124e3692688fd001e65d74 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/fd60a5ebb7124e3692688fd001e65d74 2023-03-30 03:57:20,747 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/B of b92c602cdf3a8ada8ca7fd94e7d5b443 into fd60a5ebb7124e3692688fd001e65d74(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:20,747 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:20,747 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/B, priority=12, startTime=1680148640684; duration=0sec 2023-03-30 03:57:20,748 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:20,748 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:20,749 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45250 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:20,749 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/C is initiating minor compaction (all files) 2023-03-30 03:57:20,750 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/C in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,750 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/db3c78b0ec60449399d36497487c7399, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/5454f0298f1a47e6ad2793b805ec3105, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/d93e4e8233714e39ba1065af53cca9bd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/74dc8dc94c4e40cd829460de90860860] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=44.2 K 2023-03-30 03:57:20,750 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting db3c78b0ec60449399d36497487c7399, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680148636023 2023-03-30 03:57:20,751 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5454f0298f1a47e6ad2793b805ec3105, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680148636093 2023-03-30 03:57:20,751 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting d93e4e8233714e39ba1065af53cca9bd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680148636663 2023-03-30 03:57:20,751 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 74dc8dc94c4e40cd829460de90860860, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680148638290 2023-03-30 03:57:20,782 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#C#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:20,806 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:20,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:20,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:20,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:20,828 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:57:20,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:20,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:20,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:20,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:20,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:20,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:20,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:20,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:20,842 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:20,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:20,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:20,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,843 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,843 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:20,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:20,856 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/36456ebf7c2542ee806e2269e41db5e7 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/36456ebf7c2542ee806e2269e41db5e7 2023-03-30 03:57:20,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:20,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:20,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:20,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:20,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:20,861 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/C of b92c602cdf3a8ada8ca7fd94e7d5b443 into 36456ebf7c2542ee806e2269e41db5e7(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:20,861 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:20,861 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/C, priority=12, startTime=1680148640684; duration=0sec 2023-03-30 03:57:20,862 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:20,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:20,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:20,871 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,871 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:20,871 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:20,871 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:20,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:20,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:20,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:20,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:20,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:20,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb143a3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:20,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:20,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:20,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:20,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:20,883 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,883 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:20,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,889 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:20,889 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,889 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:20,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:20,894 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:20,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:20,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:20,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:20,894 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:20,900 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,901 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,904 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:20,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:20,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:20,904 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:20,904 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:20,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:20,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,907 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:20,907 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:20,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:20,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:20,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:20,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:20,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:20,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148640904, max timestamp=1680148640924Num uniques 859; 2023-03-30 03:57:20,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:20,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148640904, max timestamp=1680148640924Num uniques 859; 2023-03-30 03:57:20,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:20,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148640904, max timestamp=1680148640924Num uniques 859; 2023-03-30 03:57:20,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:20,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:20,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148700976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:20,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:20,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148700976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,004 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:21,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:21,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:21,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:21,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:21,008 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:21,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:21,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:21,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,008 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,009 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:21,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:21,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:21,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:21,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:21,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:21,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:21,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:21,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:21,024 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,025 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,025 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:21,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50676504[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:21,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:21,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,028 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:21,028 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:21,028 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:21,028 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,028 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:21,029 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:21,029 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,029 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:21,029 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:21,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:21,031 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:21,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:21,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:21,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:21,031 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:21,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,033 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,036 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:21,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:21,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,037 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,037 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,037 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,037 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,037 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,037 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:21,037 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:21,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148701080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148701081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,125 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/6f83f1b5432d415cab9deee0c1156255 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/6f83f1b5432d415cab9deee0c1156255 2023-03-30 03:57:21,130 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/A of b92c602cdf3a8ada8ca7fd94e7d5b443 into 6f83f1b5432d415cab9deee0c1156255(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:21,131 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:21,131 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/A, priority=12, startTime=1680148640684; duration=0sec 2023-03-30 03:57:21,131 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:21,137 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:21,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:21,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:21,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:21,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:21,182 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:21,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:21,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:21,183 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,183 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,184 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,184 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,184 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:21,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:21,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:21,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:21,187 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:21,187 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,188 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:21,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:21,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:21,189 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,189 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:21,189 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:21,189 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:21,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:21,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:21,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:21,191 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,191 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,191 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,191 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:21,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:21,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:21,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f6cb10[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:21,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:21,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:21,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:21,194 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,194 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:21,195 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:21,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,195 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,195 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:21,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:21,197 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:21,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:21,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:21,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:21,197 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:21,199 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,199 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,199 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,201 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:21,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,207 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:21,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:21,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,207 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,207 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,207 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:21,207 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:21,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/49879be61181424f9e8008e1b481c815 2023-03-30 03:57:21,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:21,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148701284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:21,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148701284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,307 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:21,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:21,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:21,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/8cd1ae3c039748e2bcfa8f1065a65241 2023-03-30 03:57:21,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:21,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:21,313 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:21,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:21,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:21,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,313 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,314 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,315 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,315 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,315 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:21,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:21,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:21,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:21,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:21,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:21,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:21,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:21,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:21,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:21,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:21,325 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:21,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa5916b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:21,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:21,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,334 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:21,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:21,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:21,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,340 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:21,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:21,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:21,343 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:21,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:21,343 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:21,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,358 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,365 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,367 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,367 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:21,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,368 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:21,368 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:21,468 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:21,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:21,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:21,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:21,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:21,472 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:21,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:21,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:21,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,478 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,479 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,479 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:21,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:21,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:21,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:21,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:21,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:21,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:21,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:21,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:21,497 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:21,497 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:21,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:21,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148701497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:21,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:21,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:21,500 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:21,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e286567[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:21,500 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:21,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:21,501 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,501 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:21,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:21,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,502 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:21,502 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,502 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:21,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148701507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:21,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:21,512 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:21,512 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:21,512 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:21,512 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:21,512 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:21,516 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,516 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,518 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,561 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,561 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,566 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,566 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,566 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:21,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:21,566 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:21,566 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:21,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:21,566 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,566 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,569 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,569 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,569 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:21,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148701589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148701590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,666 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:21,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:21,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:21,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:21,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:21,671 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:21,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:21,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:21,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,673 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,673 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,673 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,673 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:21,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:21,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:21,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:21,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:21,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:21,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:21,676 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,676 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:21,676 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:21,676 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:21,677 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,679 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,679 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,679 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:21,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:21,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@550c68e3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:21,679 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:21,680 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:21,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:21,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:21,682 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,682 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:21,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:21,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:21,682 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:21,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:21,686 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:21,686 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:21,686 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:21,686 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:21,686 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:21,687 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:21,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:21,689 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,690 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,690 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,690 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:21,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,693 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:21,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:21,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:21,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:21,694 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,694 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:21,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:21,749 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/1901a7e9391e4d5e8604e2b860e7307f 2023-03-30 03:57:21,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/49879be61181424f9e8008e1b481c815 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/49879be61181424f9e8008e1b481c815 2023-03-30 03:57:21,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/49879be61181424f9e8008e1b481c815, entries=100, sequenceid=141, filesize=9.4 K 2023-03-30 03:57:21,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/8cd1ae3c039748e2bcfa8f1065a65241 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8cd1ae3c039748e2bcfa8f1065a65241 2023-03-30 03:57:21,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8cd1ae3c039748e2bcfa8f1065a65241, entries=100, sequenceid=141, filesize=9.4 K 2023-03-30 03:57:21,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/1901a7e9391e4d5e8604e2b860e7307f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/1901a7e9391e4d5e8604e2b860e7307f 2023-03-30 03:57:21,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/1901a7e9391e4d5e8604e2b860e7307f, entries=100, sequenceid=141, filesize=9.4 K 2023-03-30 03:57:21,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 941ms, sequenceid=141, compaction requested=false 2023-03-30 03:57:21,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:21,794 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:21,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:21,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:21,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:21,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:21,804 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:21,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:21,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:21,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,806 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:21,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:21,806 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:21,806 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:21,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:21,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:21,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:21,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:21,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:21,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:21,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:21,807 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:21,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:21,807 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:21,807 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 03:57:21,807 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:21,807 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:21,808 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:21,808 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:21,808 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:21,808 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:22,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:22,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148702105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148702106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148702211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148702211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,220 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/a04518d1037b4b22aabf843e280e9b58 2023-03-30 03:57:22,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148702414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148702416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148702436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,441 DEBUG [Thread-1772] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:57:22,655 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/1e75c504906a4dde9da8683c9f6af9b1 2023-03-30 03:57:22,683 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/eb0447aa670e4d818878ad9fcf9a3a98 2023-03-30 03:57:22,691 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/a04518d1037b4b22aabf843e280e9b58 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/a04518d1037b4b22aabf843e280e9b58 2023-03-30 03:57:22,699 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/a04518d1037b4b22aabf843e280e9b58, entries=150, sequenceid=180, filesize=11.8 K 2023-03-30 03:57:22,700 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/1e75c504906a4dde9da8683c9f6af9b1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1e75c504906a4dde9da8683c9f6af9b1 2023-03-30 03:57:22,706 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1e75c504906a4dde9da8683c9f6af9b1, entries=150, sequenceid=180, filesize=11.8 K 2023-03-30 03:57:22,707 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/eb0447aa670e4d818878ad9fcf9a3a98 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/eb0447aa670e4d818878ad9fcf9a3a98 2023-03-30 03:57:22,710 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/eb0447aa670e4d818878ad9fcf9a3a98, entries=150, sequenceid=180, filesize=11.8 K 2023-03-30 03:57:22,711 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 904ms, sequenceid=180, compaction requested=true 2023-03-30 03:57:22,711 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:22,711 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:22,711 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:22,711 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:22,711 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:22,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:22,719 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,719 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:22,719 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,719 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:22,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:22,720 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:57:22,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:22,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:22,720 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:22,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46b18719[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:22,721 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:22,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:22,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:22,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:22,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:22,724 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:22,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:22,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:22,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:22,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:22,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:22,726 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:22,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/f7013daa33654a24a1f64da466016eec 2023-03-30 03:57:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:22,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:22,752 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:22,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:22,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:22,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:22,752 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:22,766 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,766 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,766 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:22,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:22,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:22,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/4f43f2dc04014e399478b646939d32c6 2023-03-30 03:57:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,769 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:22,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:22,774 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:22,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:22,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:22,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:22,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:22,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:22,774 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,775 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:22,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:22,775 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:22,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:22,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148702776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148702777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,878 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:22,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:22,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:22,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:22,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148702880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,882 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:22,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:22,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:22,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:22,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:22,883 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:22,883 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:22,883 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,883 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:22,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:22,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:22,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:22,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:22,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:22,888 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:22,888 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:22,888 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:22,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:22,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:22,890 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:22,890 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:22,890 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:22,890 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:22,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:22,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:22,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:22,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148702888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:22,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:22,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,906 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:22,907 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,907 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:22,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:22,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75ac686d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:22,908 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:22,908 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,909 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,909 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:22,909 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:22,909 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:22,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:22,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,910 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:22,911 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:22,911 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:22,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:22,914 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:22,914 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:22,914 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:22,914 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:22,914 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:22,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:22,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:22,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:22,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:22,916 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,917 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:22,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:22,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:22,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:22,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:22,920 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:22,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:22,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:22,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:22,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:22,921 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,921 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:22,921 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:22,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:22,921 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:22,921 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:23,021 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:23,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:23,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:23,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:23,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:23,038 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:23,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:23,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:23,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:23,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,039 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:23,039 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:23,040 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,040 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:23,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:23,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:23,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:23,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:23,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:23,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:23,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:23,041 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,042 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:23,042 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:23,042 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:23,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:23,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,043 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,043 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,043 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:23,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:23,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d238e2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:23,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,045 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:23,045 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:23,045 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:23,045 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:23,046 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:23,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:23,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:23,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:23,048 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:23,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:23,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:23,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:23,048 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:23,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:23,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,050 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:23,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:23,066 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:23,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,069 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:23,069 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:23,070 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:23,070 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:23,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:23,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:23,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148703083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148703093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,169 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:23,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:23,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:23,187 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:23,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:23,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:23,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:23,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,188 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:23,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:23,188 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,188 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,188 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:23,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:23,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:23,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:23,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:23,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:23,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:23,190 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,190 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:23,190 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:23,190 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:23,191 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,191 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,192 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,192 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:23,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eaf026b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:23,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,193 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,193 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:23,193 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:23,193 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:23,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:23,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,195 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:23,195 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:23,195 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:23,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:23,197 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:23,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:23,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:23,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:23,197 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:23,198 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,198 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:23,198 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,199 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/01f6a1bfbc85495195d4829adb8b6128 2023-03-30 03:57:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,201 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,206 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,207 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:23,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:23,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:23,208 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:23,209 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:23,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/f7013daa33654a24a1f64da466016eec as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/f7013daa33654a24a1f64da466016eec 2023-03-30 03:57:23,215 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:23,215 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:23,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:23,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:23,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/f7013daa33654a24a1f64da466016eec, entries=150, sequenceid=192, filesize=11.8 K 2023-03-30 03:57:23,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/4f43f2dc04014e399478b646939d32c6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4f43f2dc04014e399478b646939d32c6 2023-03-30 03:57:23,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4f43f2dc04014e399478b646939d32c6, entries=150, sequenceid=192, filesize=11.8 K 2023-03-30 03:57:23,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/01f6a1bfbc85495195d4829adb8b6128 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/01f6a1bfbc85495195d4829adb8b6128 2023-03-30 03:57:23,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/01f6a1bfbc85495195d4829adb8b6128, entries=150, sequenceid=192, filesize=11.8 K 2023-03-30 03:57:23,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 520ms, sequenceid=192, compaction requested=true 2023-03-30 03:57:23,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:23,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:23,241 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:23,241 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:23,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:23,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:23,245 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45788 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:23,245 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/A is initiating minor compaction (all files) 2023-03-30 03:57:23,245 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45788 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:23,245 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/A in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,245 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/B is initiating minor compaction (all files) 2023-03-30 03:57:23,245 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/B in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,245 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/6f83f1b5432d415cab9deee0c1156255, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/49879be61181424f9e8008e1b481c815, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/a04518d1037b4b22aabf843e280e9b58, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/f7013daa33654a24a1f64da466016eec] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=44.7 K 2023-03-30 03:57:23,245 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/fd60a5ebb7124e3692688fd001e65d74, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8cd1ae3c039748e2bcfa8f1065a65241, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1e75c504906a4dde9da8683c9f6af9b1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4f43f2dc04014e399478b646939d32c6] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=44.7 K 2023-03-30 03:57:23,246 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 6f83f1b5432d415cab9deee0c1156255, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680148638290 2023-03-30 03:57:23,246 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting fd60a5ebb7124e3692688fd001e65d74, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680148638290 2023-03-30 03:57:23,246 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 49879be61181424f9e8008e1b481c815, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680148640820 2023-03-30 03:57:23,246 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting a04518d1037b4b22aabf843e280e9b58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680148640955 2023-03-30 03:57:23,248 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8cd1ae3c039748e2bcfa8f1065a65241, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680148640820 2023-03-30 03:57:23,251 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1e75c504906a4dde9da8683c9f6af9b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680148640955 2023-03-30 03:57:23,251 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting f7013daa33654a24a1f64da466016eec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680148642104 2023-03-30 03:57:23,252 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4f43f2dc04014e399478b646939d32c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680148642104 2023-03-30 03:57:23,278 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#B#compaction#237 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:23,282 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#A#compaction#238 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:23,309 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:23,309 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/10b07ecb3c8d41e289c565a89dbe69dc as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/10b07ecb3c8d41e289c565a89dbe69dc 2023-03-30 03:57:23,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:23,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:23,313 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/A of b92c602cdf3a8ada8ca7fd94e7d5b443 into 10b07ecb3c8d41e289c565a89dbe69dc(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:23,313 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:23,313 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/A, priority=12, startTime=1680148643241; duration=0sec 2023-03-30 03:57:23,313 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:23,314 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:23,314 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45788 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:23,315 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/C is initiating minor compaction (all files) 2023-03-30 03:57:23,315 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/C in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,315 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/36456ebf7c2542ee806e2269e41db5e7, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/1901a7e9391e4d5e8604e2b860e7307f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/eb0447aa670e4d818878ad9fcf9a3a98, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/01f6a1bfbc85495195d4829adb8b6128] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=44.7 K 2023-03-30 03:57:23,315 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 36456ebf7c2542ee806e2269e41db5e7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680148638290 2023-03-30 03:57:23,315 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1901a7e9391e4d5e8604e2b860e7307f, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680148640820 2023-03-30 03:57:23,315 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting eb0447aa670e4d818878ad9fcf9a3a98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680148640955 2023-03-30 03:57:23,316 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 01f6a1bfbc85495195d4829adb8b6128, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680148642104 2023-03-30 03:57:23,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:23,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:23,318 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:23,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:23,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:23,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:23,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,320 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,320 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:23,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:23,320 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,320 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:23,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:23,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:23,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:23,322 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:57:23,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:23,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:23,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:23,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:23,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:23,322 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:23,323 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#C#compaction#239 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:23,343 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/33feffa1cc454c1ab2098996b5685409 2023-03-30 03:57:23,379 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/996811c33420465c996a07c721ebcf54 2023-03-30 03:57:23,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:23,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:23,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148703410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148703410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148703514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148703514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148703514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,519 DEBUG [Thread-1776] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:57:23,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148703523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,530 DEBUG [Thread-1774] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:57:23,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148703717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:23,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148703718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,727 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/cb74491b246048f9b83b0ca4349b7f07 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/cb74491b246048f9b83b0ca4349b7f07 2023-03-30 03:57:23,732 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/B of b92c602cdf3a8ada8ca7fd94e7d5b443 into cb74491b246048f9b83b0ca4349b7f07(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:23,732 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:23,732 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/B, priority=12, startTime=1680148643241; duration=0sec 2023-03-30 03:57:23,732 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:23,762 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/72800875babc476486cf53f92e4f4895 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/72800875babc476486cf53f92e4f4895 2023-03-30 03:57:23,775 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/C of b92c602cdf3a8ada8ca7fd94e7d5b443 into 72800875babc476486cf53f92e4f4895(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:23,775 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:23,775 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/C, priority=12, startTime=1680148643241; duration=0sec 2023-03-30 03:57:23,775 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:23,805 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/b1faa09e140f4a1382fb818f8a6e0144 2023-03-30 03:57:23,810 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/33feffa1cc454c1ab2098996b5685409 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/33feffa1cc454c1ab2098996b5685409 2023-03-30 03:57:23,826 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/33feffa1cc454c1ab2098996b5685409, entries=150, sequenceid=213, filesize=11.8 K 2023-03-30 03:57:23,827 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/996811c33420465c996a07c721ebcf54 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/996811c33420465c996a07c721ebcf54 2023-03-30 03:57:23,831 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/996811c33420465c996a07c721ebcf54, entries=150, sequenceid=213, filesize=11.8 K 2023-03-30 03:57:23,832 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/b1faa09e140f4a1382fb818f8a6e0144 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b1faa09e140f4a1382fb818f8a6e0144 2023-03-30 03:57:23,843 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b1faa09e140f4a1382fb818f8a6e0144, entries=150, sequenceid=213, filesize=11.8 K 2023-03-30 03:57:23,844 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 521ms, sequenceid=213, compaction requested=false 2023-03-30 03:57:23,844 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:23,844 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,844 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:23,844 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:23,844 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,844 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:23,844 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:23,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:23,846 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:23,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ecc3e8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:23,849 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:23,849 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,849 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,850 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:23,853 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,853 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:23,853 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:23,853 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:23,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,856 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:23,856 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,856 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:23,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:23,859 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:23,859 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:23,859 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:23,859 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:23,859 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:23,860 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,860 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,860 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:23,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:23,861 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:23,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:23,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:23,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:23,872 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:23,872 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:23,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:23,872 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:23,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:23,973 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:23,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:23,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:23,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:23,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:23,982 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:23,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:23,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:23,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:23,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,983 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:23,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:23,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:23,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:23,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:23,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:23,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:23,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:23,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:23,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:23,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:23,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:23,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:23,987 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:23,987 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:23,988 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:57:23,988 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:23,988 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:23,988 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:23,988 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:23,988 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:23,988 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:24,023 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/62227fba039645da94a18f448a21bbe3 2023-03-30 03:57:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:24,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:24,062 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/ade7f85c2fb942a3864879f665574a05 2023-03-30 03:57:24,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148704064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148704065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,077 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/0cd476c394334e77817a56a516321453 2023-03-30 03:57:24,089 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/62227fba039645da94a18f448a21bbe3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/62227fba039645da94a18f448a21bbe3 2023-03-30 03:57:24,092 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/62227fba039645da94a18f448a21bbe3, entries=150, sequenceid=228, filesize=11.8 K 2023-03-30 03:57:24,093 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/ade7f85c2fb942a3864879f665574a05 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ade7f85c2fb942a3864879f665574a05 2023-03-30 03:57:24,106 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ade7f85c2fb942a3864879f665574a05, entries=150, sequenceid=228, filesize=11.8 K 2023-03-30 03:57:24,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/0cd476c394334e77817a56a516321453 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/0cd476c394334e77817a56a516321453 2023-03-30 03:57:24,112 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/0cd476c394334e77817a56a516321453, entries=150, sequenceid=228, filesize=11.8 K 2023-03-30 03:57:24,112 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 124ms, sequenceid=228, compaction requested=true 2023-03-30 03:57:24,112 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:24,113 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,114 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:24,114 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:24,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:24,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62153218[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:24,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:24,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,118 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:24,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:24,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:24,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:24,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:24,121 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:24,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:24,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:24,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:24,121 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:24,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,122 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,122 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,132 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:24,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:24,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,132 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:24,132 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:24,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:24,170 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:57:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:24,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148704184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148704187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,232 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:24,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:24,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:24,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:24,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:24,242 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:24,242 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,242 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:24,242 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:24,243 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,243 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,243 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,243 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,244 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,244 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:24,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:24,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:24,245 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:24,245 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,245 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:24,246 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:24,246 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:24,247 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,247 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:24,247 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:24,247 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,247 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:24,247 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:24,247 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,247 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:24,247 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:24,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,248 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,248 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:24,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:24,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b991390[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:24,249 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:24,249 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,254 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:24,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:24,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:24,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:24,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:24,256 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:24,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:24,258 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:24,259 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:24,259 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:24,259 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:24,259 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:24,260 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,260 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,260 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,264 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,264 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:24,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:24,264 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,264 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:24,264 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:24,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148704288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148704292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,364 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:24,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:24,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:24,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:24,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:24,378 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:24,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:24,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:24,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,385 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,385 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,385 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:24,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:24,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:24,388 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:24,388 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,388 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:24,388 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:24,388 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:24,388 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,388 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:24,388 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:24,388 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,388 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:24,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:24,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:24,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:24,389 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,389 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,390 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,390 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:24,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d63fad2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:24,391 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:24,391 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,391 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,391 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:24,391 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:24,392 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:24,391 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,392 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:24,392 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,393 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:24,393 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,393 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:24,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:24,395 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:24,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:24,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:24,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:24,395 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:24,396 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,396 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,397 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,398 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,399 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:24,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,402 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,402 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,402 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:24,402 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,402 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:24,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,402 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:24,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148704493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148704495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,502 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:24,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:24,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:24,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:24,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:24,506 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:24,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:24,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:24,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,507 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,507 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,507 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,507 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:24,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:24,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:24,508 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:24,508 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,508 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:24,509 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:24,509 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:24,509 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,509 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:24,510 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:24,510 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:24,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:24,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:24,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:24,511 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,511 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,511 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:24,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f028d15[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:24,513 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:24,513 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,514 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:24,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:24,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:24,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:24,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:24,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:24,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:24,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:24,528 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:24,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:24,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:24,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:24,528 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:24,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,533 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,533 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,534 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,537 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,544 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,544 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,544 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:24,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:24,544 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,544 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,545 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,545 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:24,545 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:24,546 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,546 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/58ad27e6003548af955d491a585b42e9 2023-03-30 03:57:24,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/4aae644fa20645838d1829778a5f2fa5 2023-03-30 03:57:24,645 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:24,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:24,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:24,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:24,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:24,649 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:24,649 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:24,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:24,651 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,651 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,663 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,663 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,663 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:24,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/9f100d68a84f4e1385a862bde764900d 2023-03-30 03:57:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:24,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:24,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:24,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:24,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,666 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:24,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/58ad27e6003548af955d491a585b42e9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/58ad27e6003548af955d491a585b42e9 2023-03-30 03:57:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:24,678 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,678 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:24,678 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:24,678 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:24,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/58ad27e6003548af955d491a585b42e9, entries=200, sequenceid=248, filesize=14.1 K 2023-03-30 03:57:24,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,680 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,680 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,680 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/4aae644fa20645838d1829778a5f2fa5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4aae644fa20645838d1829778a5f2fa5 2023-03-30 03:57:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:24,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ccef1f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:24,682 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:24,682 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,697 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,698 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,698 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,698 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:24,698 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:24,698 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:24,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:24,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:24,702 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:24,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:24,705 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:24,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:24,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:24,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:24,705 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:24,707 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,707 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,707 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,708 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:24,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,712 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:24,712 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:24,712 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:24,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4aae644fa20645838d1829778a5f2fa5, entries=150, sequenceid=248, filesize=11.8 K 2023-03-30 03:57:24,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/9f100d68a84f4e1385a862bde764900d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9f100d68a84f4e1385a862bde764900d 2023-03-30 03:57:24,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9f100d68a84f4e1385a862bde764900d, entries=150, sequenceid=248, filesize=11.8 K 2023-03-30 03:57:24,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 576ms, sequenceid=248, compaction requested=true 2023-03-30 03:57:24,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:24,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:24,746 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:24,746 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:24,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:24,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:24,748 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50808 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:24,748 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/A is initiating minor compaction (all files) 2023-03-30 03:57:24,748 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:24,748 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/B is initiating minor compaction (all files) 2023-03-30 03:57:24,748 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/B in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,748 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/cb74491b246048f9b83b0ca4349b7f07, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/996811c33420465c996a07c721ebcf54, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ade7f85c2fb942a3864879f665574a05, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4aae644fa20645838d1829778a5f2fa5] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=47.3 K 2023-03-30 03:57:24,748 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/A in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,748 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/10b07ecb3c8d41e289c565a89dbe69dc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/33feffa1cc454c1ab2098996b5685409, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/62227fba039645da94a18f448a21bbe3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/58ad27e6003548af955d491a585b42e9] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=49.6 K 2023-03-30 03:57:24,749 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting cb74491b246048f9b83b0ca4349b7f07, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680148642104 2023-03-30 03:57:24,749 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 10b07ecb3c8d41e289c565a89dbe69dc, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680148642104 2023-03-30 03:57:24,749 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 996811c33420465c996a07c721ebcf54, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680148642747 2023-03-30 03:57:24,749 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 33feffa1cc454c1ab2098996b5685409, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680148642747 2023-03-30 03:57:24,749 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ade7f85c2fb942a3864879f665574a05, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680148643402 2023-03-30 03:57:24,750 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 62227fba039645da94a18f448a21bbe3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680148643402 2023-03-30 03:57:24,750 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4aae644fa20645838d1829778a5f2fa5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148644058 2023-03-30 03:57:24,750 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 58ad27e6003548af955d491a585b42e9, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148644058 2023-03-30 03:57:24,774 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#A#compaction#249 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:24,805 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:57:24,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:24,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:24,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:24,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:24,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:24,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:24,813 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:24,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:24,815 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#B#compaction#250 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:24,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:24,818 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:24,818 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,818 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:24,818 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:24,818 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,818 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,819 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,819 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,819 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:24,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:24,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:24,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:24,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:24,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:24,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:24,838 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:24,838 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:24,850 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,851 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:24,852 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:24,853 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:24,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:24,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,874 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:24,874 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:24,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,876 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,876 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:24,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@577a6585[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:24,878 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:24,878 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,879 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:24,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:24,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:24,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:24,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:24,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:24,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:24,880 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:24,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:24,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:24,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:24,880 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:24,886 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/3e8266a312b44ff99e7954da7954b435 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3e8266a312b44ff99e7954da7954b435 2023-03-30 03:57:24,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,887 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:24,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:24,888 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:24,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/8db0fd245ae74a9095b485fa5163e112 2023-03-30 03:57:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:24,891 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/B of b92c602cdf3a8ada8ca7fd94e7d5b443 into 3e8266a312b44ff99e7954da7954b435(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:24,891 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:24,891 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/B, priority=12, startTime=1680148644746; duration=0sec 2023-03-30 03:57:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,891 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:24,891 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:24,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,893 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:24,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148704886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,894 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/C is initiating minor compaction (all files) 2023-03-30 03:57:24,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,894 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/C in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:24,894 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/72800875babc476486cf53f92e4f4895, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b1faa09e140f4a1382fb818f8a6e0144, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/0cd476c394334e77817a56a516321453, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9f100d68a84f4e1385a862bde764900d] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=47.3 K 2023-03-30 03:57:24,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148704887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,894 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 72800875babc476486cf53f92e4f4895, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680148642104 2023-03-30 03:57:24,895 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b1faa09e140f4a1382fb818f8a6e0144, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680148642747 2023-03-30 03:57:24,895 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 0cd476c394334e77817a56a516321453, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680148643402 2023-03-30 03:57:24,896 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 9f100d68a84f4e1385a862bde764900d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148644058 2023-03-30 03:57:24,898 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,898 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,898 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:24,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:24,898 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,899 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,899 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:24,899 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:24,900 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:24,900 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:24,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:24,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:24,902 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:24,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:24,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:24,903 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:24,903 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:24,910 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#C#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:24,966 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/1630f8cc42bb4965a36a687dc8d8fa3b 2023-03-30 03:57:24,970 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/4104f222582b4c9b8ea2e9d572f47ab2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/4104f222582b4c9b8ea2e9d572f47ab2 2023-03-30 03:57:24,976 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/C of b92c602cdf3a8ada8ca7fd94e7d5b443 into 4104f222582b4c9b8ea2e9d572f47ab2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:24,976 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:24,976 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/C, priority=12, startTime=1680148644746; duration=0sec 2023-03-30 03:57:24,976 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:24,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148704994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:24,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:24,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40648 deadline: 1680148704995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,003 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:25,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:25,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:25,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:25,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:25,016 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:25,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:25,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:25,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:25,017 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,017 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,017 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:25,017 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:25,018 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,018 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:25,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:25,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:25,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:25,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:25,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:25,023 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:25,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:25,024 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:25,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75edb0d2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:25,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:25,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,026 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:25,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:25,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:25,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:25,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:25,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:25,029 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:25,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:25,030 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:25,030 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:25,030 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:25,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:25,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:25,040 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:25,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:25,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:25,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:25,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:25,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:25,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,049 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:25,049 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:25,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:25,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:25,149 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:25,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:25,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:25,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:25,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:25,151 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:25,152 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:25,152 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:25,152 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:25,152 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,152 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:25,153 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,153 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,153 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:25,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:25,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:25,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:25,154 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:25,154 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:25,154 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:25,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:25,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:25,159 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:25,159 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:25,159 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:25,159 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:25,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:25,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:25,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:25,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:25,165 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,165 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,165 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,165 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:25,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e3cf15f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:25,172 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:25,172 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,173 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,173 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:25,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:25,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:25,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:25,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,183 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:25,183 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:25,183 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:25,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:25,199 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:25,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:25,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:25,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:25,199 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:25,200 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,200 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,200 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,201 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:25,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:25,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148705199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148705198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,219 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:25,220 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:25,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:25,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:25,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:25,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:25,220 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:25,220 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:25,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:25,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:25,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,226 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,280 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/635dfe3ed2354973b9c6a1d3d37f6c77 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/635dfe3ed2354973b9c6a1d3d37f6c77 2023-03-30 03:57:25,284 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/A of b92c602cdf3a8ada8ca7fd94e7d5b443 into 635dfe3ed2354973b9c6a1d3d37f6c77(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:25,285 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:25,285 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/A, priority=12, startTime=1680148644746; duration=0sec 2023-03-30 03:57:25,285 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:25,320 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:25,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:25,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:25,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:25,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:25,324 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:25,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:25,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:25,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:25,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,325 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:25,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:25,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:25,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:25,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:25,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:25,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:25,329 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:25,329 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:25,329 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:25,329 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:25,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:25,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:25,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:25,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:25,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,331 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,331 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0b4b90[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:25,333 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:25,333 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,334 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:25,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:25,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:25,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:25,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:25,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:25,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:25,336 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:25,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:25,351 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:25,351 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:25,351 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:25,351 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:25,351 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:25,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:25,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,361 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:25,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:25,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:25,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:25,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:25,365 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:25,365 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:25,368 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:25,404 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/c9386345ca104638a4248a2a37b29f15 2023-03-30 03:57:25,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/8db0fd245ae74a9095b485fa5163e112 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8db0fd245ae74a9095b485fa5163e112 2023-03-30 03:57:25,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8db0fd245ae74a9095b485fa5163e112, entries=200, sequenceid=261, filesize=14.2 K 2023-03-30 03:57:25,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/1630f8cc42bb4965a36a687dc8d8fa3b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1630f8cc42bb4965a36a687dc8d8fa3b 2023-03-30 03:57:25,433 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1630f8cc42bb4965a36a687dc8d8fa3b, entries=150, sequenceid=261, filesize=11.9 K 2023-03-30 03:57:25,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/c9386345ca104638a4248a2a37b29f15 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/c9386345ca104638a4248a2a37b29f15 2023-03-30 03:57:25,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/c9386345ca104638a4248a2a37b29f15, entries=150, sequenceid=261, filesize=11.9 K 2023-03-30 03:57:25,442 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 638ms, sequenceid=261, compaction requested=false 2023-03-30 03:57:25,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:25,468 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:25,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:25,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:25,474 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:25,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:25,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:25,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:25,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:25,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:25,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:25,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:25,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:25,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:25,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:25,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:25,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:25,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:25,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:25,479 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:25,479 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:25,479 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:57:25,479 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:25,479 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:25,480 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:25,480 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:25,480 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:25,480 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:25,506 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/9ed464f32abc43cc907c84359952dd68 2023-03-30 03:57:25,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:25,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:25,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40670 deadline: 1680148705523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148705524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,556 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/2a143e975c6b4892aa1dad8404a7a153 2023-03-30 03:57:25,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:25,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148705625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148705626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:25,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148705827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148705829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:25,989 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/abf146f6d76d438e8f366fdbaa086b71 2023-03-30 03:57:25,999 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/9ed464f32abc43cc907c84359952dd68 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/9ed464f32abc43cc907c84359952dd68 2023-03-30 03:57:26,003 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/9ed464f32abc43cc907c84359952dd68, entries=150, sequenceid=284, filesize=11.9 K 2023-03-30 03:57:26,005 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/2a143e975c6b4892aa1dad8404a7a153 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/2a143e975c6b4892aa1dad8404a7a153 2023-03-30 03:57:26,009 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/2a143e975c6b4892aa1dad8404a7a153, entries=150, sequenceid=284, filesize=11.9 K 2023-03-30 03:57:26,010 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/abf146f6d76d438e8f366fdbaa086b71 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/abf146f6d76d438e8f366fdbaa086b71 2023-03-30 03:57:26,017 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/abf146f6d76d438e8f366fdbaa086b71, entries=150, sequenceid=284, filesize=11.9 K 2023-03-30 03:57:26,018 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 539ms, sequenceid=284, compaction requested=true 2023-03-30 03:57:26,018 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:26,018 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:26,019 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,019 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,019 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:26,025 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:26,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49d959f9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:26,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:26,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,031 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:26,032 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,032 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:26,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:26,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:26,035 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:26,035 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:26,035 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:26,035 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:26,035 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:26,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,037 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,040 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:26,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:26,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,041 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:26,042 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:26,042 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:26,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:26,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:57:26,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:26,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:26,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:26,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:26,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:26,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:26,142 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:26,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:26,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:26,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:26,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:26,149 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:26,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:26,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:26,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,154 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,154 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,154 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:26,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:26,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:26,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:26,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:26,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:26,172 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:26,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51aa0b30[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:26,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:26,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,179 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,179 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,179 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:26,179 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:26,179 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:26,180 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,180 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:26,180 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,180 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:26,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:26,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148706186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,190 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:26,190 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:26,190 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:26,190 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:26,190 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:26,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148706186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,204 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,204 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,204 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,205 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,214 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,220 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,220 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,220 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:26,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:26,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,220 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,220 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,220 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:26,221 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:26,220 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:26,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148706293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148706296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,321 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:26,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:26,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:26,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:26,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:26,325 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:26,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:26,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:26,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,327 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:26,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:26,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:26,328 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:26,328 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,328 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:26,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:26,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:26,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:26,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:26,340 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:26,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7835ae53[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:26,341 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:26,341 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,342 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,342 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,342 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:26,343 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:26,343 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:26,343 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:26,343 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:26,343 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,343 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:26,345 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:26,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:26,347 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:26,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:26,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:26,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:26,347 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:26,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,355 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,357 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:26,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:26,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,358 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:26,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,358 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:26,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40658 deadline: 1680148706451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,458 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:26,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:26,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:26,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:26,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:26,467 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:26,467 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,467 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:26,467 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:26,467 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,467 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,468 DEBUG [Thread-1772] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:57:26,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,469 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:26,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:26,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:26,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:26,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:26,483 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,483 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:26,484 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:26,484 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:26,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:26,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:26,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f752d0e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:26,487 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:26,487 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,488 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,489 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:26,489 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:26,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:26,489 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:26,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:26,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:26,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:26,492 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:26,492 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:26,492 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:26,492 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:26,492 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:26,493 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,496 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:26,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:26,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,499 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:26,499 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:26,499 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148706502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,506 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,506 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,506 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148706515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/1d007587034444e4bbd1ad2f39362286 2023-03-30 03:57:26,577 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/10f020336e9f4cc1808fe00df972803d 2023-03-30 03:57:26,599 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:26,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:26,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:26,610 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:26,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:26,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:26,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:26,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:26,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:26,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:26,615 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,615 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:26,615 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:26,615 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:26,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:26,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,616 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:26,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c60b788[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:26,619 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:26,619 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,619 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,619 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:26,619 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:26,619 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:26,619 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,620 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:26,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:26,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,621 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:26,621 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:26,623 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:26,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:26,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:26,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:26,624 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:26,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,625 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,626 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,627 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,627 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,627 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:26,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:26,627 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,627 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,627 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:26,627 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,628 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:26,628 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:26,628 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,628 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,628 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,728 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:26,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:26,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:26,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:26,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:26,733 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:26,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:26,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:26,734 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,734 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,734 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,734 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:26,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:26,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:26,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:26,736 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,736 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:26,736 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:26,736 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:26,736 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:26,737 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,737 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:26,737 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:26,737 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:26,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d12d2bf[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:26,739 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:26,739 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,741 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:26,741 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:26,741 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:26,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:26,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:26,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:26,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:26,742 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:26,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:26,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:26,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:26,742 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:26,743 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,743 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,743 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,744 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,747 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,747 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,747 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:26,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:26,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:26,747 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,748 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:26,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,748 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:26,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148706804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:26,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148706820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,848 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:26,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:26,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:26,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:26,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:26,854 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:26,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:26,854 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:26,858 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,858 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,861 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,861 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,861 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:26,865 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:26,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:26,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:26,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:26,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:26,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:26,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:26,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:26,867 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,867 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:26,867 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:26,867 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:26,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:26,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:26,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:26,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:26,873 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,873 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,874 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:26,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3800ae78[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:26,878 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:26,878 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,881 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:26,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:26,881 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:26,881 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:26,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:26,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:26,882 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:26,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:26,887 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:26,887 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:26,887 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:26,887 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:26,887 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:26,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:26,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:26,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:26,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:26,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:26,892 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:26,893 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:26,893 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:26,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:26,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:26,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:26,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:26,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:26,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:26,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/16f91b3c82fe489e9e443cc4bb0baaaa 2023-03-30 03:57:26,993 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:26,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:27,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/1d007587034444e4bbd1ad2f39362286 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/1d007587034444e4bbd1ad2f39362286 2023-03-30 03:57:27,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:27,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/1d007587034444e4bbd1ad2f39362286, entries=150, sequenceid=298, filesize=11.9 K 2023-03-30 03:57:27,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/10f020336e9f4cc1808fe00df972803d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/10f020336e9f4cc1808fe00df972803d 2023-03-30 03:57:27,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:27,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:27,010 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:27,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:27,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:27,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:27,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,012 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:27,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:27,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:27,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:27,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:27,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:27,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:27,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:27,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/10f020336e9f4cc1808fe00df972803d, entries=150, sequenceid=298, filesize=11.9 K 2023-03-30 03:57:27,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/16f91b3c82fe489e9e443cc4bb0baaaa as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/16f91b3c82fe489e9e443cc4bb0baaaa 2023-03-30 03:57:27,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:27,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:27,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:27,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:27,021 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:27,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:27,021 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:27,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:27,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/16f91b3c82fe489e9e443cc4bb0baaaa, entries=150, sequenceid=298, filesize=11.9 K 2023-03-30 03:57:27,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:27,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:27,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:27,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e5be92b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:27,025 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:27,025 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,026 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:27,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:27,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:27,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:27,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,030 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 895ms, sequenceid=298, compaction requested=true 2023-03-30 03:57:27,030 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:27,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:27,030 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:27,030 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:27,031 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:27,031 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:27,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:27,031 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:27,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:27,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:27,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:27,031 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:27,032 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51394 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:27,032 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/A is initiating minor compaction (all files) 2023-03-30 03:57:27,032 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/A in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,032 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/635dfe3ed2354973b9c6a1d3d37f6c77, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8db0fd245ae74a9095b485fa5163e112, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/9ed464f32abc43cc907c84359952dd68, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/1d007587034444e4bbd1ad2f39362286] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=50.2 K 2023-03-30 03:57:27,033 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 635dfe3ed2354973b9c6a1d3d37f6c77, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148644058 2023-03-30 03:57:27,033 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 8db0fd245ae74a9095b485fa5163e112, keycount=200, bloomtype=ROW, size=14.2 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680148644174 2023-03-30 03:57:27,033 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:27,033 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/B is initiating minor compaction (all files) 2023-03-30 03:57:27,033 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/B in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,033 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3e8266a312b44ff99e7954da7954b435, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1630f8cc42bb4965a36a687dc8d8fa3b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/2a143e975c6b4892aa1dad8404a7a153, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/10f020336e9f4cc1808fe00df972803d] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=47.9 K 2023-03-30 03:57:27,034 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9ed464f32abc43cc907c84359952dd68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680148644867 2023-03-30 03:57:27,034 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3e8266a312b44ff99e7954da7954b435, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148644058 2023-03-30 03:57:27,034 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1d007587034444e4bbd1ad2f39362286, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148645522 2023-03-30 03:57:27,034 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1630f8cc42bb4965a36a687dc8d8fa3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680148644182 2023-03-30 03:57:27,035 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2a143e975c6b4892aa1dad8404a7a153, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680148644867 2023-03-30 03:57:27,035 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 10f020336e9f4cc1808fe00df972803d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148645522 2023-03-30 03:57:27,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:27,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,042 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,042 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,043 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:27,047 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:27,047 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:27,047 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:27,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:27,049 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:27,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:27,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,049 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:27,049 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:27,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:27,050 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#B#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:27,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:27,055 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#A#compaction#262 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:27,111 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/85e334bfbf984f63b3ef0544675d4027 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85e334bfbf984f63b3ef0544675d4027 2023-03-30 03:57:27,114 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/A of b92c602cdf3a8ada8ca7fd94e7d5b443 into 85e334bfbf984f63b3ef0544675d4027(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:27,114 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:27,114 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/A, priority=12, startTime=1680148647030; duration=0sec 2023-03-30 03:57:27,115 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:27,115 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:27,116 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:27,116 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/C is initiating minor compaction (all files) 2023-03-30 03:57:27,116 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/C in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,116 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/4104f222582b4c9b8ea2e9d572f47ab2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/c9386345ca104638a4248a2a37b29f15, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/abf146f6d76d438e8f366fdbaa086b71, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/16f91b3c82fe489e9e443cc4bb0baaaa] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=47.9 K 2023-03-30 03:57:27,116 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 4104f222582b4c9b8ea2e9d572f47ab2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148644058 2023-03-30 03:57:27,116 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting c9386345ca104638a4248a2a37b29f15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680148644182 2023-03-30 03:57:27,116 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting abf146f6d76d438e8f366fdbaa086b71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680148644867 2023-03-30 03:57:27,117 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 16f91b3c82fe489e9e443cc4bb0baaaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148645522 2023-03-30 03:57:27,127 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#C#compaction#263 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:27,149 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:27,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:27,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:27,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:27,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:27,156 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:27,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:27,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:27,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:27,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:27,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:27,158 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,158 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:27,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:27,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:27,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:27,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:27,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:27,160 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:27,160 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:27,160 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,160 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:27,160 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 03:57:27,161 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:27,161 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,161 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:27,161 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,161 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:27,161 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:27,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:27,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:27,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148707341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:27,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148707341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:27,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148707444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148707446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,486 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/b3d1e21f37664e4fb789287b7ee3289b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/b3d1e21f37664e4fb789287b7ee3289b 2023-03-30 03:57:27,491 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/B of b92c602cdf3a8ada8ca7fd94e7d5b443 into b3d1e21f37664e4fb789287b7ee3289b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:27,491 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:27,491 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/B, priority=12, startTime=1680148647030; duration=0sec 2023-03-30 03:57:27,491 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:27,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:27,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40710 deadline: 1680148707535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,541 DEBUG [Thread-1774] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:57:27,553 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/97e478a4798f42e09ff8d0246c7a7782 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/97e478a4798f42e09ff8d0246c7a7782 2023-03-30 03:57:27,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:27,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40686 deadline: 1680148707557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,561 DEBUG [Thread-1776] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:57:27,565 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/C of b92c602cdf3a8ada8ca7fd94e7d5b443 into 97e478a4798f42e09ff8d0246c7a7782(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:27,565 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:27,565 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/C, priority=12, startTime=1680148647031; duration=0sec 2023-03-30 03:57:27,565 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:27,603 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/59f6996e6ed34c50a44d7ccc976b588f 2023-03-30 03:57:27,616 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/897173fccb7a48e2a18611919f941cec 2023-03-30 03:57:27,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:27,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148707647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:27,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148707650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,658 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/7bfbb34cb1f945b590d4d7e573b0a4be 2023-03-30 03:57:27,673 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/59f6996e6ed34c50a44d7ccc976b588f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/59f6996e6ed34c50a44d7ccc976b588f 2023-03-30 03:57:27,678 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/59f6996e6ed34c50a44d7ccc976b588f, entries=150, sequenceid=317, filesize=11.9 K 2023-03-30 03:57:27,679 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/897173fccb7a48e2a18611919f941cec as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/897173fccb7a48e2a18611919f941cec 2023-03-30 03:57:27,685 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/897173fccb7a48e2a18611919f941cec, entries=150, sequenceid=317, filesize=11.9 K 2023-03-30 03:57:27,687 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/7bfbb34cb1f945b590d4d7e573b0a4be as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7bfbb34cb1f945b590d4d7e573b0a4be 2023-03-30 03:57:27,698 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7bfbb34cb1f945b590d4d7e573b0a4be, entries=150, sequenceid=317, filesize=11.9 K 2023-03-30 03:57:27,699 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 539ms, sequenceid=317, compaction requested=false 2023-03-30 03:57:27,699 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:27,699 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:27,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:27,699 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:27,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:27,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,703 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:27,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:27,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:27,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@176ecd9f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:27,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:27,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,707 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:27,707 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:27,707 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:27,707 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,707 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:27,708 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:27,708 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:27,708 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:27,709 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:27,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:27,712 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:27,712 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:27,712 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:27,712 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:27,712 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:27,714 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,714 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:27,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,724 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:27,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:27,725 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,726 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,727 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:27,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:27,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:27,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:27,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:27,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:27,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:27,728 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:27,728 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:27,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,828 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:27,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:27,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:27,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:27,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:27,836 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:27,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:27,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:27,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:27,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:27,837 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,837 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,837 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:27,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:27,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:27,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:27,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:27,842 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:27,842 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:27,842 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:27,847 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:27,847 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,847 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:27,847 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:27,847 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 03:57:27,848 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:27,848 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,848 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:27,848 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,848 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:27,848 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,883 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/b1b23275ae7b4391aed551ee2f772b4b 2023-03-30 03:57:27,926 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/3018fc3d0fec4f1ea2b9b02cf7b5b05f 2023-03-30 03:57:27,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:27,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:27,961 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/62ae52a3daa44aa6b728cf815cf7b5f1 2023-03-30 03:57:27,971 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/b1b23275ae7b4391aed551ee2f772b4b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b1b23275ae7b4391aed551ee2f772b4b 2023-03-30 03:57:27,975 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b1b23275ae7b4391aed551ee2f772b4b, entries=150, sequenceid=333, filesize=11.9 K 2023-03-30 03:57:27,975 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/3018fc3d0fec4f1ea2b9b02cf7b5b05f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3018fc3d0fec4f1ea2b9b02cf7b5b05f 2023-03-30 03:57:27,978 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3018fc3d0fec4f1ea2b9b02cf7b5b05f, entries=150, sequenceid=333, filesize=11.9 K 2023-03-30 03:57:27,979 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/62ae52a3daa44aa6b728cf815cf7b5f1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/62ae52a3daa44aa6b728cf815cf7b5f1 2023-03-30 03:57:27,982 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/62ae52a3daa44aa6b728cf815cf7b5f1, entries=150, sequenceid=333, filesize=11.9 K 2023-03-30 03:57:27,982 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=60.38 KB/61830 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 135ms, sequenceid=333, compaction requested=true 2023-03-30 03:57:27,982 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:27,982 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:27,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:27,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:27,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:27,986 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:27,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,987 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:27,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:27,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:27,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:57:27,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:27,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:27,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c1272e7[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:27,994 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:27,994 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:27,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:27,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:27,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:27,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:27,999 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:27,999 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:27,999 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:27,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:27,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:28,000 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:28,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:28,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:28,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:28,003 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:28,003 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:28,003 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:28,003 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:28,003 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:28,004 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,004 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:28,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:28,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,006 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,006 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:28,010 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:28,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:28,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:28,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:28,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:28,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,011 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:28,011 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:28,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148708030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148708038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,111 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:28,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:28,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:28,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:28,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:28,114 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:28,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:28,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:28,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:28,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,116 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:28,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:28,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:28,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:28,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:28,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:28,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:28,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:28,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:28,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:28,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:28,121 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:28,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515c6273[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:28,125 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:28,125 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:28,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:28,125 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:28,126 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,126 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:28,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:28,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:28,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:28,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:28,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:28,136 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:28,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:28,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:28,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:28,136 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:28,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148708135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:28,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:28,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:28,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,145 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148708142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:28,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:28,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:28,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:28,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:28,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:28,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:28,157 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:28,157 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:28,257 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:28,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:28,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:28,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:28,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:28,267 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:28,267 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:28,268 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:28,268 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:28,268 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,269 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,269 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:28,270 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:28,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:28,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:28,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:28,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:28,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:28,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:28,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,278 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:28,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70efcbc2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:28,287 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:28,287 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,288 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:28,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:28,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:28,289 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:28,289 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:28,289 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:28,293 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,294 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:28,294 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:28,294 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:28,295 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:28,295 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:28,295 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:28,295 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:28,306 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,306 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,306 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,311 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,311 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:28,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:28,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:28,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:28,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,314 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:28,314 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:28,314 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:28,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:28,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:28,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148708338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148708347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,415 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:28,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/ffbd26f155904a20984a0d5a319d5ea1 2023-03-30 03:57:28,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:28,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:28,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:28,425 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:28,425 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:28,425 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:28,425 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:28,425 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,425 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,426 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,426 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,426 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:28,426 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:28,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:28,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:28,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:28,427 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:28,427 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:28,427 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:28,427 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:28,427 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:28,427 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,428 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:28,428 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:28,428 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,428 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:28,428 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:28,428 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,428 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:28,428 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:28,429 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,429 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,429 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,429 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:28,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@110ce275[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:28,431 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:28,431 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,432 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,432 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:28,432 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,432 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:28,432 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:28,432 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:28,433 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,433 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:28,433 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:28,433 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:28,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:28,435 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:28,435 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:28,436 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:28,436 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:28,436 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:28,436 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,436 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:28,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,437 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:28,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,437 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,438 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:28,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:28,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:28,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:28,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:28,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:28,441 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,441 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:28,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:28,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:28,441 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:28,441 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:28,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/567921bc954d4614b02f6b59d2c64851 2023-03-30 03:57:28,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/720e386a16144bce8ca126e9c35abb59 2023-03-30 03:57:28,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/ffbd26f155904a20984a0d5a319d5ea1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/ffbd26f155904a20984a0d5a319d5ea1 2023-03-30 03:57:28,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/ffbd26f155904a20984a0d5a319d5ea1, entries=150, sequenceid=347, filesize=11.9 K 2023-03-30 03:57:28,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/567921bc954d4614b02f6b59d2c64851 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/567921bc954d4614b02f6b59d2c64851 2023-03-30 03:57:28,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/567921bc954d4614b02f6b59d2c64851, entries=150, sequenceid=347, filesize=11.9 K 2023-03-30 03:57:28,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/720e386a16144bce8ca126e9c35abb59 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/720e386a16144bce8ca126e9c35abb59 2023-03-30 03:57:28,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/720e386a16144bce8ca126e9c35abb59, entries=150, sequenceid=347, filesize=11.9 K 2023-03-30 03:57:28,488 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 494ms, sequenceid=347, compaction requested=true 2023-03-30 03:57:28,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:28,488 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:28,488 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:28,488 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:28,488 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:28,488 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:28,498 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:28,498 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:28,498 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/A is initiating minor compaction (all files) 2023-03-30 03:57:28,499 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/B is initiating minor compaction (all files) 2023-03-30 03:57:28,499 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/A in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,499 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85e334bfbf984f63b3ef0544675d4027, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/59f6996e6ed34c50a44d7ccc976b588f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b1b23275ae7b4391aed551ee2f772b4b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/ffbd26f155904a20984a0d5a319d5ea1] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=48.1 K 2023-03-30 03:57:28,499 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/B in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,499 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/b3d1e21f37664e4fb789287b7ee3289b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/897173fccb7a48e2a18611919f941cec, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3018fc3d0fec4f1ea2b9b02cf7b5b05f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/567921bc954d4614b02f6b59d2c64851] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=48.1 K 2023-03-30 03:57:28,499 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 85e334bfbf984f63b3ef0544675d4027, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148645522 2023-03-30 03:57:28,510 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b3d1e21f37664e4fb789287b7ee3289b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148645522 2023-03-30 03:57:28,510 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 59f6996e6ed34c50a44d7ccc976b588f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680148646183 2023-03-30 03:57:28,510 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 897173fccb7a48e2a18611919f941cec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680148646183 2023-03-30 03:57:28,511 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting b1b23275ae7b4391aed551ee2f772b4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680148647336 2023-03-30 03:57:28,511 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3018fc3d0fec4f1ea2b9b02cf7b5b05f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680148647336 2023-03-30 03:57:28,511 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting ffbd26f155904a20984a0d5a319d5ea1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148647957 2023-03-30 03:57:28,511 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 567921bc954d4614b02f6b59d2c64851, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148647957 2023-03-30 03:57:28,517 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#B#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:28,517 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#A#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:28,541 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:28,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:28,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:28,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:28,554 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:28,554 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:28,554 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:28,554 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:28,554 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,554 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:28,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:28,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:28,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:28,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:28,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:28,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:28,560 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:28,560 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:28,560 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:28,562 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:28,562 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:28,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:28,563 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 03:57:28,563 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:28,563 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:28,563 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:28,563 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:28,563 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:28,563 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:28,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:28,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148708691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148708692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148708796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:28,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148708797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:28,970 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/beb00fe24a624f0d806206921b6f3217 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/beb00fe24a624f0d806206921b6f3217 2023-03-30 03:57:28,974 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/c79d69d12a244dafbc106212914d6f10 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c79d69d12a244dafbc106212914d6f10 2023-03-30 03:57:28,976 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/A of b92c602cdf3a8ada8ca7fd94e7d5b443 into beb00fe24a624f0d806206921b6f3217(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:28,976 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:28,976 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/A, priority=12, startTime=1680148648488; duration=0sec 2023-03-30 03:57:28,976 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:28,976 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:28,979 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:28,979 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/C is initiating minor compaction (all files) 2023-03-30 03:57:28,979 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/C in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:28,979 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/97e478a4798f42e09ff8d0246c7a7782, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7bfbb34cb1f945b590d4d7e573b0a4be, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/62ae52a3daa44aa6b728cf815cf7b5f1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/720e386a16144bce8ca126e9c35abb59] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=48.1 K 2023-03-30 03:57:28,979 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/B of b92c602cdf3a8ada8ca7fd94e7d5b443 into c79d69d12a244dafbc106212914d6f10(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:28,979 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:28,980 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/B, priority=12, startTime=1680148648488; duration=0sec 2023-03-30 03:57:28,980 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 97e478a4798f42e09ff8d0246c7a7782, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680148645522 2023-03-30 03:57:28,980 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:28,980 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 7bfbb34cb1f945b590d4d7e573b0a4be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680148646183 2023-03-30 03:57:28,981 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 62ae52a3daa44aa6b728cf815cf7b5f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1680148647336 2023-03-30 03:57:28,982 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 720e386a16144bce8ca126e9c35abb59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148647957 2023-03-30 03:57:28,999 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#C#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:29,001 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/5225247e76294b6ab65120a9aaa40421 2023-03-30 03:57:29,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148708999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148709001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,031 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/25a69f02cd2f47efb1c0a2f6ea86b793 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/25a69f02cd2f47efb1c0a2f6ea86b793 2023-03-30 03:57:29,035 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/C of b92c602cdf3a8ada8ca7fd94e7d5b443 into 25a69f02cd2f47efb1c0a2f6ea86b793(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:29,035 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:29,035 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/C, priority=12, startTime=1680148648488; duration=0sec 2023-03-30 03:57:29,035 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:29,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:29,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148709304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:29,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148709305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,444 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/5b0ea06302ce498db183fa89946d37de 2023-03-30 03:57:29,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:29,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148709806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:29,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148709812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,860 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/b42c8d21f55f40ee9402f177a38e8ca0 2023-03-30 03:57:29,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/5225247e76294b6ab65120a9aaa40421 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/5225247e76294b6ab65120a9aaa40421 2023-03-30 03:57:29,870 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/5225247e76294b6ab65120a9aaa40421, entries=150, sequenceid=365, filesize=11.9 K 2023-03-30 03:57:29,871 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/5b0ea06302ce498db183fa89946d37de as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/5b0ea06302ce498db183fa89946d37de 2023-03-30 03:57:29,874 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/5b0ea06302ce498db183fa89946d37de, entries=150, sequenceid=365, filesize=11.9 K 2023-03-30 03:57:29,882 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/b42c8d21f55f40ee9402f177a38e8ca0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b42c8d21f55f40ee9402f177a38e8ca0 2023-03-30 03:57:29,886 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b42c8d21f55f40ee9402f177a38e8ca0, entries=150, sequenceid=365, filesize=11.9 K 2023-03-30 03:57:29,887 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 1324ms, sequenceid=365, compaction requested=false 2023-03-30 03:57:29,887 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:29,887 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:29,887 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:29,887 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:29,887 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,887 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:29,887 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:29,888 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,888 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:29,888 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:29,888 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:29,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:29,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e762ff9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:29,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:29,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:29,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:29,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:29,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:29,891 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:29,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:29,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:29,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:29,893 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:29,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:29,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:29,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:29,893 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:29,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:29,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:29,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:29,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:29,902 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:29,904 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,907 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:29,907 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:29,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:29,907 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:29,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:29,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:29,908 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:29,908 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:29,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:29,912 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,912 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:29,912 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:29,913 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:29,913 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:29,913 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:29,913 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,008 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:30,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:30,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:30,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:30,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:30,011 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:30,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:30,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:30,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:30,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,012 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:30,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:30,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:30,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:30,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:30,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:30,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:30,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:30,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:30,016 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:30,016 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:30,016 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:30,016 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:30,016 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 03:57:30,016 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:30,016 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:30,016 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:30,016 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:30,016 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:30,017 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:30,061 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/0c2d295d7f124e1fa61ac43608181b48 2023-03-30 03:57:30,106 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/1baab926e12f4deeb39b3816cf84abcc 2023-03-30 03:57:30,543 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/890069e2acb34ab2a9524d0f1347c31f 2023-03-30 03:57:30,546 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/0c2d295d7f124e1fa61ac43608181b48 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/0c2d295d7f124e1fa61ac43608181b48 2023-03-30 03:57:30,549 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/0c2d295d7f124e1fa61ac43608181b48, entries=150, sequenceid=382, filesize=11.9 K 2023-03-30 03:57:30,550 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/1baab926e12f4deeb39b3816cf84abcc as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1baab926e12f4deeb39b3816cf84abcc 2023-03-30 03:57:30,553 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1baab926e12f4deeb39b3816cf84abcc, entries=150, sequenceid=382, filesize=11.9 K 2023-03-30 03:57:30,553 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/890069e2acb34ab2a9524d0f1347c31f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/890069e2acb34ab2a9524d0f1347c31f 2023-03-30 03:57:30,562 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/890069e2acb34ab2a9524d0f1347c31f, entries=150, sequenceid=382, filesize=11.9 K 2023-03-30 03:57:30,563 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 547ms, sequenceid=382, compaction requested=true 2023-03-30 03:57:30,563 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:30,563 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:30,563 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:30,563 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:30,563 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,563 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:30,563 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:30,567 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:30,569 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:30,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8b87be[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:30,569 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,569 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:30,570 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:30,570 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:30,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:30,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:30,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:30,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:30,584 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:30,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:30,587 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:30,587 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:30,587 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:30,587 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:30,587 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:30,588 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,588 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:30,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:30,588 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,589 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,592 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:30,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:30,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:30,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:30,592 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:30,592 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:30,592 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:30,592 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:30,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:30,593 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,593 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:30,593 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,598 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,692 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:30,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:30,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:30,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:30,700 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:30,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:30,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:30,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:30,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,701 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,701 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:30,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:30,701 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,701 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:30,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:30,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:30,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:30,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:30,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:30,716 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:30,717 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:30,717 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:30,717 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:30,717 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:30,717 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:30,717 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,717 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:30,717 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:30,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,723 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,724 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:30,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b4e636[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:30,725 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:30,725 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,726 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:30,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:30,726 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:30,726 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:30,727 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,727 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,727 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:30,727 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:30,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:30,729 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:30,729 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:30,729 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:30,729 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:30,729 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:30,730 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,730 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:30,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:30,731 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,735 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:30,735 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:30,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:30,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:30,735 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,735 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,735 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,736 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:30,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:30,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:30,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:30,736 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,738 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,738 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,738 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,738 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:30,738 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:30,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:30,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:57:30,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:30,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:30,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:30,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:30,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:30,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:30,840 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:30,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:30,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:30,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:30,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:30,846 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:30,846 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:30,846 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:30,846 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:30,846 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,847 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,847 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:30,847 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,848 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,848 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:30,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:30,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:30,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:30,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:30,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:30,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:30,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:30,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:30,868 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:30,868 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:30,868 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:30,868 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:30,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:30,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:30,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:30,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:30,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/04a0f149d34643c28910d73882626dc0 2023-03-30 03:57:30,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:30,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148650857, max timestamp=1680148650884Num uniques 859; 2023-03-30 03:57:30,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148650857, max timestamp=1680148650884Num uniques 859; 2023-03-30 03:57:30,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:57:30,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:57:30,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148650857, max timestamp=1680148650884Num uniques 859; 2023-03-30 03:57:30,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:57:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:30,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24dda95f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:30,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:30,890 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,891 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:30,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:30,891 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:30,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:30,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:30,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:30,892 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,892 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:30,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:30,893 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:30,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:30,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:30,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:30,893 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:30,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:30,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:30,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,895 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:30,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:30,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:30,897 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:30,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:30,897 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:30,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:30,897 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:30,897 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:30,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:30,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:30,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148710930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:30,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148710930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:30,997 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:30,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:30,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:31,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:31,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:31,003 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:31,003 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:31,003 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:31,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:31,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,009 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:31,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:31,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:31,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:31,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:31,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:31,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:31,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:31,012 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:31,012 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,012 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:31,013 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:31,013 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,013 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:31,013 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:31,013 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,013 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:31,013 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:31,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:31,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ef81495[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:31,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:31,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:31,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,014 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,017 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:31,024 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:31,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:31,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:31,025 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,025 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:31,025 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:31,025 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:31,025 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:31,025 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:31,025 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:31,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:31,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:31,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:31,026 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:31,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:31,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,027 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:31,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:31,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,031 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:31,031 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:31,032 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:31,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:31,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:31,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,032 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:31,032 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:31,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148711036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148711036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,132 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:31,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:31,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:31,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:31,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:31,135 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:31,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:31,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:31,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:31,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,135 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,136 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:31,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:31,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:31,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,137 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:31,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:31,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:31,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:31,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:31,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:31,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:31,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:31,143 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,144 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:31,144 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:31,144 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:31,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,148 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:31,149 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,149 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:31,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:31,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b8b53da[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:31,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:31,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,155 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,155 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:31,158 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:31,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:31,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:31,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:31,159 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:31,159 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:31,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:31,163 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:31,163 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:31,163 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:31,163 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:31,163 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:31,164 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,165 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:31,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:31,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,169 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:31,169 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:31,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:31,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:31,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:31,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,172 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:31,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:31,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:31,172 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:31,172 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:31,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:31,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148711239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148711239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,272 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:31,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:31,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:31,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:31,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:31,286 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:31,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:31,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:31,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:31,287 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,287 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,287 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:31,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:31,288 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,288 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:31,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:31,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:31,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:31,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:31,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:31,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:31,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:31,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:31,300 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:31,301 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:31,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,302 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:31,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/ca0a2c13b19042bcbd0d97697ff94d1b 2023-03-30 03:57:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:31,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:31,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3d8d62[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:31,304 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:31,304 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:31,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:31,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:31,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:31,315 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:31,315 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:31,315 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:31,315 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/85d8a6fd48044a979dbe1383125e1424 2023-03-30 03:57:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:31,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:31,318 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:31,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:31,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:31,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:31,318 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:31,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/04a0f149d34643c28910d73882626dc0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/04a0f149d34643c28910d73882626dc0 2023-03-30 03:57:31,322 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,322 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:31,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:31,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,322 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:31,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:31,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:31,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:31,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/04a0f149d34643c28910d73882626dc0, entries=250, sequenceid=394, filesize=16.7 K 2023-03-30 03:57:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,326 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/ca0a2c13b19042bcbd0d97697ff94d1b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ca0a2c13b19042bcbd0d97697ff94d1b 2023-03-30 03:57:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:31,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:31,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:31,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:31,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:31,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:31,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:31,332 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:31,332 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:31,332 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:31,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:31,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:31,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ca0a2c13b19042bcbd0d97697ff94d1b, entries=150, sequenceid=394, filesize=11.9 K 2023-03-30 03:57:31,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/85d8a6fd48044a979dbe1383125e1424 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/85d8a6fd48044a979dbe1383125e1424 2023-03-30 03:57:31,353 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/85d8a6fd48044a979dbe1383125e1424, entries=150, sequenceid=394, filesize=11.9 K 2023-03-30 03:57:31,353 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 519ms, sequenceid=394, compaction requested=true 2023-03-30 03:57:31,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:31,354 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:31,354 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:31,354 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:31,356 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:31,356 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/B is initiating minor compaction (all files) 2023-03-30 03:57:31,356 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/B in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,356 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c79d69d12a244dafbc106212914d6f10, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/5b0ea06302ce498db183fa89946d37de, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1baab926e12f4deeb39b3816cf84abcc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ca0a2c13b19042bcbd0d97697ff94d1b] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=48.3 K 2023-03-30 03:57:31,356 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54306 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:31,356 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/A is initiating minor compaction (all files) 2023-03-30 03:57:31,356 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/A in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,356 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/beb00fe24a624f0d806206921b6f3217, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/5225247e76294b6ab65120a9aaa40421, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/0c2d295d7f124e1fa61ac43608181b48, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/04a0f149d34643c28910d73882626dc0] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=53.0 K 2023-03-30 03:57:31,357 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting beb00fe24a624f0d806206921b6f3217, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148647957 2023-03-30 03:57:31,358 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c79d69d12a244dafbc106212914d6f10, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148647957 2023-03-30 03:57:31,358 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5225247e76294b6ab65120a9aaa40421, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1680148648022 2023-03-30 03:57:31,358 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 0c2d295d7f124e1fa61ac43608181b48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680148648657 2023-03-30 03:57:31,359 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 04a0f149d34643c28910d73882626dc0, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680148650828 2023-03-30 03:57:31,360 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:31,361 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5b0ea06302ce498db183fa89946d37de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1680148648022 2023-03-30 03:57:31,363 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1baab926e12f4deeb39b3816cf84abcc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680148648657 2023-03-30 03:57:31,363 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ca0a2c13b19042bcbd0d97697ff94d1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680148650828 2023-03-30 03:57:31,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:31,376 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#B#compaction#285 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:31,391 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#A#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:31,433 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:31,448 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/e88c53f531da4355808b0705af74ee4c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/e88c53f531da4355808b0705af74ee4c 2023-03-30 03:57:31,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:31,454 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:31,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:31,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:31,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:31,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:31,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:31,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:31,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,456 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,456 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:31,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:31,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:31,457 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/B of b92c602cdf3a8ada8ca7fd94e7d5b443 into e88c53f531da4355808b0705af74ee4c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:31,457 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:31,457 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/B, priority=12, startTime=1680148651354; duration=0sec 2023-03-30 03:57:31,457 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:31,457 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:31,459 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:31,459 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/C is initiating minor compaction (all files) 2023-03-30 03:57:31,459 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/C in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,459 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/25a69f02cd2f47efb1c0a2f6ea86b793, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b42c8d21f55f40ee9402f177a38e8ca0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/890069e2acb34ab2a9524d0f1347c31f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/85d8a6fd48044a979dbe1383125e1424] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=48.3 K 2023-03-30 03:57:31,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:31,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:31,459 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:31,459 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:31,459 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:31,459 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 25a69f02cd2f47efb1c0a2f6ea86b793, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680148647957 2023-03-30 03:57:31,460 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b42c8d21f55f40ee9402f177a38e8ca0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1680148648022 2023-03-30 03:57:31,460 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 890069e2acb34ab2a9524d0f1347c31f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680148648657 2023-03-30 03:57:31,460 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 85d8a6fd48044a979dbe1383125e1424, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680148650828 2023-03-30 03:57:31,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:31,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:31,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:31,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:31,466 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-30 03:57:31,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:31,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:31,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:31,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:31,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:31,467 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:31,479 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#C#compaction#287 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:31,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:31,546 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:31,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148711565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148711566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148711671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148711671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,846 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/b84919bef3bd4dbc983876fc40fb4744 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b84919bef3bd4dbc983876fc40fb4744 2023-03-30 03:57:31,850 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/A of b92c602cdf3a8ada8ca7fd94e7d5b443 into b84919bef3bd4dbc983876fc40fb4744(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:31,850 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:31,850 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/A, priority=12, startTime=1680148651354; duration=0sec 2023-03-30 03:57:31,850 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:31,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148711878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:31,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148711878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:31,926 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/85dda13a9ca04cf1908f40553c56fb3c 2023-03-30 03:57:31,942 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/35afebaacbf74905a3c7488def7bcd23 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/35afebaacbf74905a3c7488def7bcd23 2023-03-30 03:57:31,948 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/C of b92c602cdf3a8ada8ca7fd94e7d5b443 into 35afebaacbf74905a3c7488def7bcd23(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:31,948 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:31,948 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/C, priority=12, startTime=1680148651367; duration=0sec 2023-03-30 03:57:31,948 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:32,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:32,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148712183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:32,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148712186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,369 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/519b8e58244f408098090b44b708c675 2023-03-30 03:57:32,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:32,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148712693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:32,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148712694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,792 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/9e3c56a86e3f4ca58e825bb230c0129e 2023-03-30 03:57:32,797 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/85dda13a9ca04cf1908f40553c56fb3c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85dda13a9ca04cf1908f40553c56fb3c 2023-03-30 03:57:32,800 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85dda13a9ca04cf1908f40553c56fb3c, entries=150, sequenceid=431, filesize=11.9 K 2023-03-30 03:57:32,801 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/519b8e58244f408098090b44b708c675 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/519b8e58244f408098090b44b708c675 2023-03-30 03:57:32,803 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/519b8e58244f408098090b44b708c675, entries=150, sequenceid=431, filesize=11.9 K 2023-03-30 03:57:32,804 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/9e3c56a86e3f4ca58e825bb230c0129e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9e3c56a86e3f4ca58e825bb230c0129e 2023-03-30 03:57:32,817 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9e3c56a86e3f4ca58e825bb230c0129e, entries=150, sequenceid=431, filesize=11.9 K 2023-03-30 03:57:32,818 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 1352ms, sequenceid=431, compaction requested=false 2023-03-30 03:57:32,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:32,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:32,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:32,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:32,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:32,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:32,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:32,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:32,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:32,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:32,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:32,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ecd4165[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:32,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:32,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:32,831 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:32,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:32,832 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:32,832 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:32,832 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:32,832 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,832 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:32,833 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:32,833 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,833 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:32,833 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:32,835 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:32,835 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:32,835 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:32,835 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:32,835 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:32,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:32,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:32,837 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:32,843 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:32,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:32,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,850 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:32,851 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:32,851 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,851 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:32,851 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:32,851 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,851 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:32,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:32,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:32,851 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:32,851 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:32,951 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:32,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:32,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:32,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:32,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:32,954 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:32,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:32,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:32,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:32,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,954 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:32,955 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:32,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:32,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:32,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:32,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:32,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:32,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:32,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:32,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:32,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:32,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:32,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:32,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:32,962 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:32,962 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:32,962 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:57:32,962 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:32,962 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:32,962 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:32,962 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:32,962 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:32,962 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:33,394 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/eca8f7944fa34ba1a61a4bc0282a5124 2023-03-30 03:57:33,428 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/1b78f099a4b74a35b315ab2fd5996e65 2023-03-30 03:57:33,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:33,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:33,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148653733, max timestamp=1680148653742Num uniques 859; 2023-03-30 03:57:33,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148653733, max timestamp=1680148653742Num uniques 859; 2023-03-30 03:57:33,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:33,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148653733, max timestamp=1680148653742Num uniques 859; 2023-03-30 03:57:33,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:33,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:57:33,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:33,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148713773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:33,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148713774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,852 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/e540231d921243778c3ad0ef36d0dcc2 2023-03-30 03:57:33,860 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/eca8f7944fa34ba1a61a4bc0282a5124 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/eca8f7944fa34ba1a61a4bc0282a5124 2023-03-30 03:57:33,863 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/eca8f7944fa34ba1a61a4bc0282a5124, entries=150, sequenceid=443, filesize=11.9 K 2023-03-30 03:57:33,864 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/1b78f099a4b74a35b315ab2fd5996e65 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1b78f099a4b74a35b315ab2fd5996e65 2023-03-30 03:57:33,866 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1b78f099a4b74a35b315ab2fd5996e65, entries=150, sequenceid=443, filesize=11.9 K 2023-03-30 03:57:33,867 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/e540231d921243778c3ad0ef36d0dcc2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/e540231d921243778c3ad0ef36d0dcc2 2023-03-30 03:57:33,870 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/e540231d921243778c3ad0ef36d0dcc2, entries=150, sequenceid=443, filesize=11.9 K 2023-03-30 03:57:33,871 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 909ms, sequenceid=443, compaction requested=true 2023-03-30 03:57:33,871 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:33,871 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:33,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:33,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:33,871 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,874 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:33,874 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:33,878 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,878 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:33,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:33,879 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:33,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:33,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@471396b2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:33,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:33,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:33,881 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:33,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:33,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:33,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:33,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:33,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,882 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:33,882 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:33,882 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:33,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:33,883 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680148653759, max timestamp=1680148653881Num uniques 1018; 2023-03-30 03:57:33,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 03:57:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:33,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:33,885 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:33,885 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:33,885 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:33,885 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:33,885 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:33,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680148653759, max timestamp=1680148653881Num uniques 1018; 2023-03-30 03:57:33,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 03:57:33,886 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:33,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680148653759, max timestamp=1680148653881Num uniques 1018; 2023-03-30 03:57:33,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:33,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 03:57:33,886 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:33,886 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:33,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:33,887 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:33,888 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:33,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:33,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:33,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:33,890 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:33,891 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:33,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:33,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:33,891 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,891 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:33,891 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:33,891 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:33,891 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:33,891 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:33,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:33,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:33,897 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-03-30 03:57:33,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:33,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:33,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:33,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:33,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:33,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:33,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148653915, max timestamp=1680148653945Num uniques 859; 2023-03-30 03:57:33,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:33,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148653915, max timestamp=1680148653945Num uniques 859; 2023-03-30 03:57:33,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:33,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148653915, max timestamp=1680148653945Num uniques 859; 2023-03-30 03:57:33,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:33,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:33,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148713985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:33,991 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:33,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:33,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:33,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:33,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148713990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,014 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,014 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,015 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,017 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,017 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,017 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,022 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,022 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,022 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,022 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,022 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,023 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,023 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,024 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,025 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66a71809[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,025 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,025 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,026 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,026 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,026 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,027 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,027 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,029 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,029 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,030 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,030 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,030 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,030 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,034 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,034 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:34,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148714092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:34,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148714098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,134 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:34,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:34,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:34,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,139 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,139 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,139 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,139 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,140 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,140 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,141 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,141 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,141 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,141 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,142 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,142 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,142 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,142 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,142 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,142 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,143 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,143 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,144 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,144 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71e6df22[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,144 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,145 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,145 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,145 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,145 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,146 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,146 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,146 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,146 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,148 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,148 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,148 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,148 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,148 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,150 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,152 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:34,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,153 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,153 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,152 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,154 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,253 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:34,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:34,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:34,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,256 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,257 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,257 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,257 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,257 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,258 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,258 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,258 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,258 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,259 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,260 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,260 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c47e12[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,261 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,262 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,262 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,263 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,263 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,263 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,263 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,265 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,265 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,266 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,267 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,269 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,269 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,269 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,269 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:34,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148714297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:34,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148714302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/8646611d71ff4a8a9b520f66e74edd63 2023-03-30 03:57:34,370 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:34,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:34,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:34,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,373 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,373 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,374 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,374 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,374 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,374 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,375 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,375 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,376 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,376 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,376 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,376 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,376 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,377 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,377 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e39cc1e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,378 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,378 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,378 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,378 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,379 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,379 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,379 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,379 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,381 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,381 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,381 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,381 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,381 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,385 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,385 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,385 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,485 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:34,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:34,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:34,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,489 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,489 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,492 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,492 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,492 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,492 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,493 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,493 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,493 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,493 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1000e280[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,495 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,495 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,495 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,495 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,496 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,496 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,496 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,498 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,498 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,499 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,499 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,500 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,502 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,501 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:34,502 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,502 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,602 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:34,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:34,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:34,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,605 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,606 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,606 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,606 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,606 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:34,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148714603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:34,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148714608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,612 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,612 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,612 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,613 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,622 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,622 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,623 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,623 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5758206d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,625 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,625 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,625 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,625 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,625 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,627 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,627 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,627 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,627 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,627 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,629 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,629 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,629 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,629 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,630 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:34,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,632 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,632 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,632 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,632 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,732 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:34,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:34,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:34,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/a535352bab574c859ed77c6b6c3bfbb8 2023-03-30 03:57:34,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,735 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,735 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,736 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,736 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,736 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,736 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,738 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,738 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61dae07a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,741 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,741 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,742 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,742 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,743 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,745 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,745 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,745 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,745 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,745 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,745 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,745 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,745 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,749 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,749 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,849 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:34,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:34,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,852 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,852 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,852 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,852 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,852 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,852 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,853 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,853 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,853 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,853 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,854 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,854 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,854 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,854 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,856 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,856 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61838963[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,857 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,857 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,857 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,857 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,858 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,858 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,858 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,860 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,860 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,860 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,860 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,860 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,861 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,861 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,861 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:34,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,864 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,864 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,964 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:34,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:34,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:34,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:34,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:34,967 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:34,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:34,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:34,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,968 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,968 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,968 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:34,970 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:34,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:34,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:34,971 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,971 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:34,971 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:34,971 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:34,972 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,972 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,972 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:34,973 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:34,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76007331[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:34,973 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,974 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,974 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:34,974 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:34,974 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:34,974 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,974 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:34,975 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:34,977 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:34,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:34,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:34,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:34,977 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:34,978 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,978 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:34,978 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:34,978 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,979 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,981 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:34,981 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,981 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:34,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:34,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:34,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:34,981 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:34,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:34,981 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,982 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:34,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:34,982 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:34,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:34,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:34,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,082 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:35,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:35,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:35,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:35,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:35,084 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:35,084 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:35,084 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:35,084 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:35,085 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,085 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:35,085 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,085 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:35,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:35,086 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,086 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:35,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:35,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:35,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:35,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:35,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:35,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:35,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:35,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:35,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. as already flushing 2023-03-30 03:57:35,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:35,088 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:35,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:35,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:35,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@604d7ed4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:35,089 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:35,089 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:35,089 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:35,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:35,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:35,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:35,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:35,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:35,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:35,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:35,091 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:35,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:35,092 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:35,092 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:35,092 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:35,092 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:35,092 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:35,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,093 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:35,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:35,093 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:35,094 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:35,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:35,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:35,096 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:35,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:35,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:35,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:35,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:35,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,096 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:35,096 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:35,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:35,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:35,097 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:35,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:35,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40648 deadline: 1680148715113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:35,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40670 deadline: 1680148715113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/28d97ab12bc7415ab498bb7f04beff5f 2023-03-30 03:57:35,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/8646611d71ff4a8a9b520f66e74edd63 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8646611d71ff4a8a9b520f66e74edd63 2023-03-30 03:57:35,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8646611d71ff4a8a9b520f66e74edd63, entries=200, sequenceid=486, filesize=14.3 K 2023-03-30 03:57:35,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/a535352bab574c859ed77c6b6c3bfbb8 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a535352bab574c859ed77c6b6c3bfbb8 2023-03-30 03:57:35,162 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a535352bab574c859ed77c6b6c3bfbb8, entries=150, sequenceid=486, filesize=11.9 K 2023-03-30 03:57:35,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/28d97ab12bc7415ab498bb7f04beff5f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/28d97ab12bc7415ab498bb7f04beff5f 2023-03-30 03:57:35,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/28d97ab12bc7415ab498bb7f04beff5f, entries=150, sequenceid=486, filesize=11.9 K 2023-03-30 03:57:35,167 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~164.95 KB/168912, currentSize=127.47 KB/130530 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 1270ms, sequenceid=486, compaction requested=true 2023-03-30 03:57:35,167 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:35,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:35,167 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:35,167 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:35,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:35,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:35,168 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:35,168 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:35,168 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/A is initiating minor compaction (all files) 2023-03-30 03:57:35,168 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/B is initiating minor compaction (all files) 2023-03-30 03:57:35,168 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/A in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:35,168 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/B in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:35,169 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b84919bef3bd4dbc983876fc40fb4744, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85dda13a9ca04cf1908f40553c56fb3c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/eca8f7944fa34ba1a61a4bc0282a5124, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8646611d71ff4a8a9b520f66e74edd63] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=50.8 K 2023-03-30 03:57:35,169 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/e88c53f531da4355808b0705af74ee4c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/519b8e58244f408098090b44b708c675, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1b78f099a4b74a35b315ab2fd5996e65, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a535352bab574c859ed77c6b6c3bfbb8] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=48.4 K 2023-03-30 03:57:35,169 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting b84919bef3bd4dbc983876fc40fb4744, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680148650828 2023-03-30 03:57:35,169 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e88c53f531da4355808b0705af74ee4c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680148650828 2023-03-30 03:57:35,169 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 85dda13a9ca04cf1908f40553c56fb3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1680148650908 2023-03-30 03:57:35,169 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 519b8e58244f408098090b44b708c675, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1680148650908 2023-03-30 03:57:35,169 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting eca8f7944fa34ba1a61a4bc0282a5124, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680148651556 2023-03-30 03:57:35,170 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1b78f099a4b74a35b315ab2fd5996e65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680148651556 2023-03-30 03:57:35,170 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 8646611d71ff4a8a9b520f66e74edd63, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680148653885 2023-03-30 03:57:35,170 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a535352bab574c859ed77c6b6c3bfbb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680148653894 2023-03-30 03:57:35,176 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#A#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:35,176 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#B#compaction#298 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:35,194 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/2327a23c05a84557b9c964945caf649c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2327a23c05a84557b9c964945caf649c 2023-03-30 03:57:35,197 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:35,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:35,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:35,199 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/A of b92c602cdf3a8ada8ca7fd94e7d5b443 into 2327a23c05a84557b9c964945caf649c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:35,199 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:35,199 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/A, priority=12, startTime=1680148655167; duration=0sec 2023-03-30 03:57:35,199 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:35,199 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:35,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:35,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:35,200 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:35,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:35,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:35,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:35,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:35,201 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:35,201 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,201 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:35,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:35,201 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): b92c602cdf3a8ada8ca7fd94e7d5b443/C is initiating minor compaction (all files) 2023-03-30 03:57:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:35,201 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b92c602cdf3a8ada8ca7fd94e7d5b443/C in TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:35,201 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/35afebaacbf74905a3c7488def7bcd23, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9e3c56a86e3f4ca58e825bb230c0129e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/e540231d921243778c3ad0ef36d0dcc2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/28d97ab12bc7415ab498bb7f04beff5f] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp, totalSize=48.4 K 2023-03-30 03:57:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:35,201 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:35,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:35,202 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 35afebaacbf74905a3c7488def7bcd23, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680148650828 2023-03-30 03:57:35,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:35,202 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9e3c56a86e3f4ca58e825bb230c0129e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1680148650908 2023-03-30 03:57:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:35,202 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting e540231d921243778c3ad0ef36d0dcc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680148651556 2023-03-30 03:57:35,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:35,203 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 28d97ab12bc7415ab498bb7f04beff5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680148653894 2023-03-30 03:57:35,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:35,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:35,218 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:35,218 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:35,218 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:35,218 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:35,218 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:35,218 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:35,219 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. started... 2023-03-30 03:57:35,219 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-30 03:57:35,219 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:35,219 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:35,219 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:35,219 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:35,219 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:35,219 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:35,220 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): b92c602cdf3a8ada8ca7fd94e7d5b443#C#compaction#299 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:35,234 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/574447a9d656454fb298d853e2f74e5f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/574447a9d656454fb298d853e2f74e5f 2023-03-30 03:57:35,238 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/C of b92c602cdf3a8ada8ca7fd94e7d5b443 into 574447a9d656454fb298d853e2f74e5f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:35,238 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:35,238 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/C, priority=12, startTime=1680148655167; duration=0sec 2023-03-30 03:57:35,238 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:35,593 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/fa8a1b49bf9c43d39759fe0ea70bbb9b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/fa8a1b49bf9c43d39759fe0ea70bbb9b 2023-03-30 03:57:35,597 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b92c602cdf3a8ada8ca7fd94e7d5b443/B of b92c602cdf3a8ada8ca7fd94e7d5b443 into fa8a1b49bf9c43d39759fe0ea70bbb9b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:35,597 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:35,597 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443., storeName=b92c602cdf3a8ada8ca7fd94e7d5b443/B, priority=12, startTime=1680148655167; duration=0sec 2023-03-30 03:57:35,597 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:35,634 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/302d7d68f9d74a628adb511d25b4b366 2023-03-30 03:57:35,672 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/a7f0bbcda3cc404a83b5ffbf71a0b358 2023-03-30 03:57:35,995 DEBUG [Thread-1789] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69b9b3fd to 127.0.0.1:58033 2023-03-30 03:57:35,995 DEBUG [Thread-1789] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:35,995 DEBUG [Thread-1781] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7abacd49 to 127.0.0.1:58033 2023-03-30 03:57:35,996 DEBUG [Thread-1781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:35,997 DEBUG [Thread-1783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57d25552 to 127.0.0.1:58033 2023-03-30 03:57:35,997 DEBUG [Thread-1783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:35,997 DEBUG [Thread-1787] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0993cc7d to 127.0.0.1:58033 2023-03-30 03:57:35,997 DEBUG [Thread-1787] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:35,997 DEBUG [Thread-1785] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13695730 to 127.0.0.1:58033 2023-03-30 03:57:35,997 DEBUG [Thread-1785] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:36,087 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/f474b5f7f84d44b591e4ae4dfb4275ba 2023-03-30 03:57:36,096 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/302d7d68f9d74a628adb511d25b4b366 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/302d7d68f9d74a628adb511d25b4b366 2023-03-30 03:57:36,101 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/302d7d68f9d74a628adb511d25b4b366, entries=150, sequenceid=521, filesize=11.9 K 2023-03-30 03:57:36,102 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/a7f0bbcda3cc404a83b5ffbf71a0b358 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a7f0bbcda3cc404a83b5ffbf71a0b358 2023-03-30 03:57:36,107 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a7f0bbcda3cc404a83b5ffbf71a0b358, entries=150, sequenceid=521, filesize=11.9 K 2023-03-30 03:57:36,108 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/f474b5f7f84d44b591e4ae4dfb4275ba as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/f474b5f7f84d44b591e4ae4dfb4275ba 2023-03-30 03:57:36,112 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/f474b5f7f84d44b591e4ae4dfb4275ba, entries=150, sequenceid=521, filesize=11.9 K 2023-03-30 03:57:36,113 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 894ms, sequenceid=521, compaction requested=false 2023-03-30 03:57:36,113 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:36,113 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:36,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:36,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:36,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:36,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:36,114 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,114 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:36,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:36,115 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:36,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:36,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:36,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56228b97[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:36,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:36,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:36,117 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:36,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:36,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:36,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:36,117 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:36,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:36,118 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:36,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:36,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:36,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:36,121 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:36,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:36,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:36,121 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:36,121 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:36,122 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:36,122 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:36,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:36,122 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:36,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:36,122 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:36,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:36,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:36,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:36,125 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:36,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:36,125 DEBUG [Thread-1780] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:36,125 DEBUG [Thread-1780] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:36,125 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:36,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:36,135 DEBUG [Thread-1778] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b7f716c to 127.0.0.1:58033 2023-03-30 03:57:36,135 DEBUG [Thread-1770] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12394d27 to 127.0.0.1:58033 2023-03-30 03:57:36,135 DEBUG [Thread-1778] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:36,135 DEBUG [Thread-1770] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:36,225 DEBUG [Thread-1780] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:36,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:36,476 DEBUG [Thread-1772] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29a548ac to 127.0.0.1:58033 2023-03-30 03:57:36,476 DEBUG [Thread-1772] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:37,579 DEBUG [Thread-1776] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e3efec6 to 127.0.0.1:58033 2023-03-30 03:57:37,579 DEBUG [Thread-1776] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:37,635 DEBUG [Thread-1774] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43184c57 to 127.0.0.1:58033 2023-03-30 03:57:37,635 DEBUG [Thread-1774] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:37,635 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 03:57:37,635 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-03-30 03:57:37,635 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-03-30 03:57:37,636 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-03-30 03:57:37,636 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-03-30 03:57:37,636 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 172 2023-03-30 03:57:37,636 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 03:57:37,636 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4676 2023-03-30 03:57:37,636 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4495 2023-03-30 03:57:37,636 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4376 2023-03-30 03:57:37,636 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4691 2023-03-30 03:57:37,637 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4559 2023-03-30 03:57:37,637 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 03:57:37,637 INFO [Listener at localhost.localdomain/37215] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 03:57:37,637 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x136e067d to 127.0.0.1:58033 2023-03-30 03:57:37,637 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:57:37,639 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 03:57:37,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-30 03:57:37,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:37,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 03:57:37,646 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148657646"}]},"ts":"1680148657646"} 2023-03-30 03:57:37,647 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 03:57:37,656 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 03:57:37,656 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b92c602cdf3a8ada8ca7fd94e7d5b443, UNASSIGN}] 2023-03-30 03:57:37,658 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b92c602cdf3a8ada8ca7fd94e7d5b443, UNASSIGN 2023-03-30 03:57:37,658 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=b92c602cdf3a8ada8ca7fd94e7d5b443, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:37,658 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148657658"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148657658"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148657658"}]},"ts":"1680148657658"} 2023-03-30 03:57:37,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:57:37,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 03:57:37,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:37,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing b92c602cdf3a8ada8ca7fd94e7d5b443, disabling compactions & flushes 2023-03-30 03:57:37,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:37,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:37,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. after waiting 0 ms 2023-03-30 03:57:37,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:37,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing b92c602cdf3a8ada8ca7fd94e7d5b443 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 03:57:37,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=A 2023-03-30 03:57:37,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:37,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=B 2023-03-30 03:57:37,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:37,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b92c602cdf3a8ada8ca7fd94e7d5b443, store=C 2023-03-30 03:57:37,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:37,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 03:57:38,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/17fe4740f07441e1a97a01229b03843e 2023-03-30 03:57:38,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 03:57:38,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/9c75e49f7d9d49fda5a095427f904b22 2023-03-30 03:57:38,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 03:57:39,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/7fcd0d7bf405451299516ede2bb94d9e 2023-03-30 03:57:39,074 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/A/17fe4740f07441e1a97a01229b03843e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17fe4740f07441e1a97a01229b03843e 2023-03-30 03:57:39,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17fe4740f07441e1a97a01229b03843e, entries=150, sequenceid=531, filesize=11.9 K 2023-03-30 03:57:39,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/B/9c75e49f7d9d49fda5a095427f904b22 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9c75e49f7d9d49fda5a095427f904b22 2023-03-30 03:57:39,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9c75e49f7d9d49fda5a095427f904b22, entries=150, sequenceid=531, filesize=11.9 K 2023-03-30 03:57:39,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/.tmp/C/7fcd0d7bf405451299516ede2bb94d9e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7fcd0d7bf405451299516ede2bb94d9e 2023-03-30 03:57:39,093 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7fcd0d7bf405451299516ede2bb94d9e, entries=150, sequenceid=531, filesize=11.9 K 2023-03-30 03:57:39,093 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for b92c602cdf3a8ada8ca7fd94e7d5b443 in 1281ms, sequenceid=531, compaction requested=true 2023-03-30 03:57:39,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/45945dadeae7402ab3d748628711a40e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17d5be5bc5cf461087c43f19f77dbb59, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2cc8e79a9fe8434d8db4e485752b4e4e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/102a6174871c48b49086924af76af8bc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/6f83f1b5432d415cab9deee0c1156255, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/49879be61181424f9e8008e1b481c815, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/a04518d1037b4b22aabf843e280e9b58, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/10b07ecb3c8d41e289c565a89dbe69dc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/f7013daa33654a24a1f64da466016eec, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/33feffa1cc454c1ab2098996b5685409, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/62227fba039645da94a18f448a21bbe3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/58ad27e6003548af955d491a585b42e9, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/635dfe3ed2354973b9c6a1d3d37f6c77, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8db0fd245ae74a9095b485fa5163e112, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/9ed464f32abc43cc907c84359952dd68, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85e334bfbf984f63b3ef0544675d4027, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/1d007587034444e4bbd1ad2f39362286, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/59f6996e6ed34c50a44d7ccc976b588f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b1b23275ae7b4391aed551ee2f772b4b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/beb00fe24a624f0d806206921b6f3217, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/ffbd26f155904a20984a0d5a319d5ea1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/5225247e76294b6ab65120a9aaa40421, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/0c2d295d7f124e1fa61ac43608181b48, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/04a0f149d34643c28910d73882626dc0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b84919bef3bd4dbc983876fc40fb4744, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85dda13a9ca04cf1908f40553c56fb3c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/eca8f7944fa34ba1a61a4bc0282a5124, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8646611d71ff4a8a9b520f66e74edd63] to archive 2023-03-30 03:57:39,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:57:39,116 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/45945dadeae7402ab3d748628711a40e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/45945dadeae7402ab3d748628711a40e 2023-03-30 03:57:39,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17d5be5bc5cf461087c43f19f77dbb59 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17d5be5bc5cf461087c43f19f77dbb59 2023-03-30 03:57:39,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2cc8e79a9fe8434d8db4e485752b4e4e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2cc8e79a9fe8434d8db4e485752b4e4e 2023-03-30 03:57:39,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/102a6174871c48b49086924af76af8bc to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/102a6174871c48b49086924af76af8bc 2023-03-30 03:57:39,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/6f83f1b5432d415cab9deee0c1156255 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/6f83f1b5432d415cab9deee0c1156255 2023-03-30 03:57:39,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/49879be61181424f9e8008e1b481c815 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/49879be61181424f9e8008e1b481c815 2023-03-30 03:57:39,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/a04518d1037b4b22aabf843e280e9b58 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/a04518d1037b4b22aabf843e280e9b58 2023-03-30 03:57:39,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/10b07ecb3c8d41e289c565a89dbe69dc to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/10b07ecb3c8d41e289c565a89dbe69dc 2023-03-30 03:57:39,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/f7013daa33654a24a1f64da466016eec to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/f7013daa33654a24a1f64da466016eec 2023-03-30 03:57:39,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/33feffa1cc454c1ab2098996b5685409 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/33feffa1cc454c1ab2098996b5685409 2023-03-30 03:57:39,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/62227fba039645da94a18f448a21bbe3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/62227fba039645da94a18f448a21bbe3 2023-03-30 03:57:39,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/58ad27e6003548af955d491a585b42e9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/58ad27e6003548af955d491a585b42e9 2023-03-30 03:57:39,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/635dfe3ed2354973b9c6a1d3d37f6c77 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/635dfe3ed2354973b9c6a1d3d37f6c77 2023-03-30 03:57:39,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8db0fd245ae74a9095b485fa5163e112 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8db0fd245ae74a9095b485fa5163e112 2023-03-30 03:57:39,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/9ed464f32abc43cc907c84359952dd68 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/9ed464f32abc43cc907c84359952dd68 2023-03-30 03:57:39,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85e334bfbf984f63b3ef0544675d4027 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85e334bfbf984f63b3ef0544675d4027 2023-03-30 03:57:39,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/1d007587034444e4bbd1ad2f39362286 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/1d007587034444e4bbd1ad2f39362286 2023-03-30 03:57:39,129 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/59f6996e6ed34c50a44d7ccc976b588f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/59f6996e6ed34c50a44d7ccc976b588f 2023-03-30 03:57:39,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b1b23275ae7b4391aed551ee2f772b4b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b1b23275ae7b4391aed551ee2f772b4b 2023-03-30 03:57:39,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/beb00fe24a624f0d806206921b6f3217 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/beb00fe24a624f0d806206921b6f3217 2023-03-30 03:57:39,132 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/ffbd26f155904a20984a0d5a319d5ea1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/ffbd26f155904a20984a0d5a319d5ea1 2023-03-30 03:57:39,133 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/5225247e76294b6ab65120a9aaa40421 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/5225247e76294b6ab65120a9aaa40421 2023-03-30 03:57:39,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/0c2d295d7f124e1fa61ac43608181b48 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/0c2d295d7f124e1fa61ac43608181b48 2023-03-30 03:57:39,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/04a0f149d34643c28910d73882626dc0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/04a0f149d34643c28910d73882626dc0 2023-03-30 03:57:39,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b84919bef3bd4dbc983876fc40fb4744 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/b84919bef3bd4dbc983876fc40fb4744 2023-03-30 03:57:39,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85dda13a9ca04cf1908f40553c56fb3c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/85dda13a9ca04cf1908f40553c56fb3c 2023-03-30 03:57:39,138 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/eca8f7944fa34ba1a61a4bc0282a5124 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/eca8f7944fa34ba1a61a4bc0282a5124 2023-03-30 03:57:39,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8646611d71ff4a8a9b520f66e74edd63 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/8646611d71ff4a8a9b520f66e74edd63 2023-03-30 03:57:39,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8976e58c66964e96bb7590e95ca88836, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c4e2338479a4454da08a7034471e7046, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/852f9008b2164e9db43529c43f0230a6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/fd60a5ebb7124e3692688fd001e65d74, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9b80e95288aa44819f436869e7fe3a46, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8cd1ae3c039748e2bcfa8f1065a65241, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1e75c504906a4dde9da8683c9f6af9b1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/cb74491b246048f9b83b0ca4349b7f07, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4f43f2dc04014e399478b646939d32c6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/996811c33420465c996a07c721ebcf54, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ade7f85c2fb942a3864879f665574a05, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3e8266a312b44ff99e7954da7954b435, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4aae644fa20645838d1829778a5f2fa5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1630f8cc42bb4965a36a687dc8d8fa3b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/2a143e975c6b4892aa1dad8404a7a153, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/b3d1e21f37664e4fb789287b7ee3289b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/10f020336e9f4cc1808fe00df972803d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/897173fccb7a48e2a18611919f941cec, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3018fc3d0fec4f1ea2b9b02cf7b5b05f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c79d69d12a244dafbc106212914d6f10, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/567921bc954d4614b02f6b59d2c64851, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/5b0ea06302ce498db183fa89946d37de, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1baab926e12f4deeb39b3816cf84abcc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/e88c53f531da4355808b0705af74ee4c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ca0a2c13b19042bcbd0d97697ff94d1b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/519b8e58244f408098090b44b708c675, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1b78f099a4b74a35b315ab2fd5996e65, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a535352bab574c859ed77c6b6c3bfbb8] to archive 2023-03-30 03:57:39,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:57:39,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8976e58c66964e96bb7590e95ca88836 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8976e58c66964e96bb7590e95ca88836 2023-03-30 03:57:39,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c4e2338479a4454da08a7034471e7046 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c4e2338479a4454da08a7034471e7046 2023-03-30 03:57:39,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/852f9008b2164e9db43529c43f0230a6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/852f9008b2164e9db43529c43f0230a6 2023-03-30 03:57:39,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/fd60a5ebb7124e3692688fd001e65d74 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/fd60a5ebb7124e3692688fd001e65d74 2023-03-30 03:57:39,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9b80e95288aa44819f436869e7fe3a46 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9b80e95288aa44819f436869e7fe3a46 2023-03-30 03:57:39,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8cd1ae3c039748e2bcfa8f1065a65241 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/8cd1ae3c039748e2bcfa8f1065a65241 2023-03-30 03:57:39,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1e75c504906a4dde9da8683c9f6af9b1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1e75c504906a4dde9da8683c9f6af9b1 2023-03-30 03:57:39,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/cb74491b246048f9b83b0ca4349b7f07 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/cb74491b246048f9b83b0ca4349b7f07 2023-03-30 03:57:39,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4f43f2dc04014e399478b646939d32c6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4f43f2dc04014e399478b646939d32c6 2023-03-30 03:57:39,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/996811c33420465c996a07c721ebcf54 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/996811c33420465c996a07c721ebcf54 2023-03-30 03:57:39,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ade7f85c2fb942a3864879f665574a05 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ade7f85c2fb942a3864879f665574a05 2023-03-30 03:57:39,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3e8266a312b44ff99e7954da7954b435 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3e8266a312b44ff99e7954da7954b435 2023-03-30 03:57:39,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4aae644fa20645838d1829778a5f2fa5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/4aae644fa20645838d1829778a5f2fa5 2023-03-30 03:57:39,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1630f8cc42bb4965a36a687dc8d8fa3b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1630f8cc42bb4965a36a687dc8d8fa3b 2023-03-30 03:57:39,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/2a143e975c6b4892aa1dad8404a7a153 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/2a143e975c6b4892aa1dad8404a7a153 2023-03-30 03:57:39,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/b3d1e21f37664e4fb789287b7ee3289b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/b3d1e21f37664e4fb789287b7ee3289b 2023-03-30 03:57:39,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/10f020336e9f4cc1808fe00df972803d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/10f020336e9f4cc1808fe00df972803d 2023-03-30 03:57:39,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/897173fccb7a48e2a18611919f941cec to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/897173fccb7a48e2a18611919f941cec 2023-03-30 03:57:39,163 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3018fc3d0fec4f1ea2b9b02cf7b5b05f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/3018fc3d0fec4f1ea2b9b02cf7b5b05f 2023-03-30 03:57:39,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c79d69d12a244dafbc106212914d6f10 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/c79d69d12a244dafbc106212914d6f10 2023-03-30 03:57:39,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/567921bc954d4614b02f6b59d2c64851 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/567921bc954d4614b02f6b59d2c64851 2023-03-30 03:57:39,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/5b0ea06302ce498db183fa89946d37de to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/5b0ea06302ce498db183fa89946d37de 2023-03-30 03:57:39,168 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1baab926e12f4deeb39b3816cf84abcc to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1baab926e12f4deeb39b3816cf84abcc 2023-03-30 03:57:39,169 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/e88c53f531da4355808b0705af74ee4c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/e88c53f531da4355808b0705af74ee4c 2023-03-30 03:57:39,170 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ca0a2c13b19042bcbd0d97697ff94d1b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/ca0a2c13b19042bcbd0d97697ff94d1b 2023-03-30 03:57:39,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/519b8e58244f408098090b44b708c675 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/519b8e58244f408098090b44b708c675 2023-03-30 03:57:39,172 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1b78f099a4b74a35b315ab2fd5996e65 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/1b78f099a4b74a35b315ab2fd5996e65 2023-03-30 03:57:39,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a535352bab574c859ed77c6b6c3bfbb8 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a535352bab574c859ed77c6b6c3bfbb8 2023-03-30 03:57:39,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/db3c78b0ec60449399d36497487c7399, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/5454f0298f1a47e6ad2793b805ec3105, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/d93e4e8233714e39ba1065af53cca9bd, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/36456ebf7c2542ee806e2269e41db5e7, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/74dc8dc94c4e40cd829460de90860860, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/1901a7e9391e4d5e8604e2b860e7307f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/eb0447aa670e4d818878ad9fcf9a3a98, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/72800875babc476486cf53f92e4f4895, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/01f6a1bfbc85495195d4829adb8b6128, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b1faa09e140f4a1382fb818f8a6e0144, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/0cd476c394334e77817a56a516321453, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/4104f222582b4c9b8ea2e9d572f47ab2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9f100d68a84f4e1385a862bde764900d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/c9386345ca104638a4248a2a37b29f15, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/abf146f6d76d438e8f366fdbaa086b71, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/97e478a4798f42e09ff8d0246c7a7782, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/16f91b3c82fe489e9e443cc4bb0baaaa, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7bfbb34cb1f945b590d4d7e573b0a4be, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/62ae52a3daa44aa6b728cf815cf7b5f1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/25a69f02cd2f47efb1c0a2f6ea86b793, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/720e386a16144bce8ca126e9c35abb59, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b42c8d21f55f40ee9402f177a38e8ca0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/890069e2acb34ab2a9524d0f1347c31f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/35afebaacbf74905a3c7488def7bcd23, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/85d8a6fd48044a979dbe1383125e1424, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9e3c56a86e3f4ca58e825bb230c0129e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/e540231d921243778c3ad0ef36d0dcc2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/28d97ab12bc7415ab498bb7f04beff5f] to archive 2023-03-30 03:57:39,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:57:39,179 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/db3c78b0ec60449399d36497487c7399 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/db3c78b0ec60449399d36497487c7399 2023-03-30 03:57:39,180 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/5454f0298f1a47e6ad2793b805ec3105 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/5454f0298f1a47e6ad2793b805ec3105 2023-03-30 03:57:39,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/d93e4e8233714e39ba1065af53cca9bd to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/d93e4e8233714e39ba1065af53cca9bd 2023-03-30 03:57:39,183 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/36456ebf7c2542ee806e2269e41db5e7 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/36456ebf7c2542ee806e2269e41db5e7 2023-03-30 03:57:39,184 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/74dc8dc94c4e40cd829460de90860860 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/74dc8dc94c4e40cd829460de90860860 2023-03-30 03:57:39,185 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/1901a7e9391e4d5e8604e2b860e7307f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/1901a7e9391e4d5e8604e2b860e7307f 2023-03-30 03:57:39,186 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/eb0447aa670e4d818878ad9fcf9a3a98 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/eb0447aa670e4d818878ad9fcf9a3a98 2023-03-30 03:57:39,188 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/72800875babc476486cf53f92e4f4895 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/72800875babc476486cf53f92e4f4895 2023-03-30 03:57:39,189 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/01f6a1bfbc85495195d4829adb8b6128 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/01f6a1bfbc85495195d4829adb8b6128 2023-03-30 03:57:39,190 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b1faa09e140f4a1382fb818f8a6e0144 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b1faa09e140f4a1382fb818f8a6e0144 2023-03-30 03:57:39,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/0cd476c394334e77817a56a516321453 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/0cd476c394334e77817a56a516321453 2023-03-30 03:57:39,192 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/4104f222582b4c9b8ea2e9d572f47ab2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/4104f222582b4c9b8ea2e9d572f47ab2 2023-03-30 03:57:39,193 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9f100d68a84f4e1385a862bde764900d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9f100d68a84f4e1385a862bde764900d 2023-03-30 03:57:39,194 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/c9386345ca104638a4248a2a37b29f15 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/c9386345ca104638a4248a2a37b29f15 2023-03-30 03:57:39,195 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/abf146f6d76d438e8f366fdbaa086b71 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/abf146f6d76d438e8f366fdbaa086b71 2023-03-30 03:57:39,196 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/97e478a4798f42e09ff8d0246c7a7782 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/97e478a4798f42e09ff8d0246c7a7782 2023-03-30 03:57:39,197 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/16f91b3c82fe489e9e443cc4bb0baaaa to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/16f91b3c82fe489e9e443cc4bb0baaaa 2023-03-30 03:57:39,198 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7bfbb34cb1f945b590d4d7e573b0a4be to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7bfbb34cb1f945b590d4d7e573b0a4be 2023-03-30 03:57:39,199 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/62ae52a3daa44aa6b728cf815cf7b5f1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/62ae52a3daa44aa6b728cf815cf7b5f1 2023-03-30 03:57:39,200 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/25a69f02cd2f47efb1c0a2f6ea86b793 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/25a69f02cd2f47efb1c0a2f6ea86b793 2023-03-30 03:57:39,201 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/720e386a16144bce8ca126e9c35abb59 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/720e386a16144bce8ca126e9c35abb59 2023-03-30 03:57:39,202 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b42c8d21f55f40ee9402f177a38e8ca0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/b42c8d21f55f40ee9402f177a38e8ca0 2023-03-30 03:57:39,203 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/890069e2acb34ab2a9524d0f1347c31f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/890069e2acb34ab2a9524d0f1347c31f 2023-03-30 03:57:39,203 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/35afebaacbf74905a3c7488def7bcd23 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/35afebaacbf74905a3c7488def7bcd23 2023-03-30 03:57:39,204 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/85d8a6fd48044a979dbe1383125e1424 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/85d8a6fd48044a979dbe1383125e1424 2023-03-30 03:57:39,205 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9e3c56a86e3f4ca58e825bb230c0129e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/9e3c56a86e3f4ca58e825bb230c0129e 2023-03-30 03:57:39,206 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/e540231d921243778c3ad0ef36d0dcc2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/e540231d921243778c3ad0ef36d0dcc2 2023-03-30 03:57:39,206 DEBUG [StoreCloser-TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/28d97ab12bc7415ab498bb7f04beff5f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/28d97ab12bc7415ab498bb7f04beff5f 2023-03-30 03:57:39,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/recovered.edits/534.seqid, newMaxSeqId=534, maxSeqId=1 2023-03-30 03:57:39,211 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443. 2023-03-30 03:57:39,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for b92c602cdf3a8ada8ca7fd94e7d5b443: 2023-03-30 03:57:39,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:39,212 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=b92c602cdf3a8ada8ca7fd94e7d5b443, regionState=CLOSED 2023-03-30 03:57:39,212 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148659212"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148659212"}]},"ts":"1680148659212"} 2023-03-30 03:57:39,214 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-30 03:57:39,214 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure b92c602cdf3a8ada8ca7fd94e7d5b443, server=jenkins-hbase20.apache.org,41157,1680148568851 in 1.5540 sec 2023-03-30 03:57:39,215 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-30 03:57:39,215 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b92c602cdf3a8ada8ca7fd94e7d5b443, UNASSIGN in 1.5580 sec 2023-03-30 03:57:39,215 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148659215"}]},"ts":"1680148659215"} 2023-03-30 03:57:39,216 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 03:57:39,217 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 03:57:39,218 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5770 sec 2023-03-30 03:57:39,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 03:57:39,754 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-30 03:57:39,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-30 03:57:39,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:39,762 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:39,763 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:39,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-30 03:57:39,767 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:39,770 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/recovered.edits] 2023-03-30 03:57:39,774 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17fe4740f07441e1a97a01229b03843e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/17fe4740f07441e1a97a01229b03843e 2023-03-30 03:57:39,776 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2327a23c05a84557b9c964945caf649c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/2327a23c05a84557b9c964945caf649c 2023-03-30 03:57:39,777 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/302d7d68f9d74a628adb511d25b4b366 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/A/302d7d68f9d74a628adb511d25b4b366 2023-03-30 03:57:39,779 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9c75e49f7d9d49fda5a095427f904b22 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/9c75e49f7d9d49fda5a095427f904b22 2023-03-30 03:57:39,780 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a7f0bbcda3cc404a83b5ffbf71a0b358 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/a7f0bbcda3cc404a83b5ffbf71a0b358 2023-03-30 03:57:39,782 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/fa8a1b49bf9c43d39759fe0ea70bbb9b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/B/fa8a1b49bf9c43d39759fe0ea70bbb9b 2023-03-30 03:57:39,784 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/574447a9d656454fb298d853e2f74e5f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/574447a9d656454fb298d853e2f74e5f 2023-03-30 03:57:39,785 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7fcd0d7bf405451299516ede2bb94d9e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/7fcd0d7bf405451299516ede2bb94d9e 2023-03-30 03:57:39,787 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/f474b5f7f84d44b591e4ae4dfb4275ba to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/C/f474b5f7f84d44b591e4ae4dfb4275ba 2023-03-30 03:57:39,789 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/recovered.edits/534.seqid to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443/recovered.edits/534.seqid 2023-03-30 03:57:39,790 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/b92c602cdf3a8ada8ca7fd94e7d5b443 2023-03-30 03:57:39,790 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:57:39,793 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:39,794 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 03:57:39,796 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 03:57:39,797 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:39,797 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 03:57:39,797 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680148659797"}]},"ts":"9223372036854775807"} 2023-03-30 03:57:39,799 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 03:57:39,799 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b92c602cdf3a8ada8ca7fd94e7d5b443, NAME => 'TestAcidGuarantees,,1680148633778.b92c602cdf3a8ada8ca7fd94e7d5b443.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 03:57:39,799 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 03:57:39,799 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680148659799"}]},"ts":"9223372036854775807"} 2023-03-30 03:57:39,801 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 03:57:39,802 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:39,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 47 msec 2023-03-30 03:57:39,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-30 03:57:39,865 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-30 03:57:39,884 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 228), OpenFileDescriptor=554 (was 561), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=491 (was 355) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=10196 (was 10224) 2023-03-30 03:57:39,893 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=554, MaxFileDescriptor=60000, SystemLoadAverage=491, ProcessCount=166, AvailableMemoryMB=10196 2023-03-30 03:57:39,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:57:39,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:57:39,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:39,898 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 03:57:39,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-30 03:57:39,899 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 03:57:39,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 03:57:39,900 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:39,901 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 empty. 2023-03-30 03:57:39,901 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:39,901 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:57:40,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 03:57:40,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 03:57:40,310 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:57:40,314 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 97a07dc2687acd6de72ca851295ed5b2, NAME => 'TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp 2023-03-30 03:57:40,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 03:57:40,724 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:57:40,725 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 97a07dc2687acd6de72ca851295ed5b2, disabling compactions & flushes 2023-03-30 03:57:40,725 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:40,725 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:40,725 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. after waiting 0 ms 2023-03-30 03:57:40,725 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:40,725 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:40,725 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:40,731 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 03:57:40,733 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148660733"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148660733"}]},"ts":"1680148660733"} 2023-03-30 03:57:40,735 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 03:57:40,736 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 03:57:40,736 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148660736"}]},"ts":"1680148660736"} 2023-03-30 03:57:40,737 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 03:57:40,740 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, ASSIGN}] 2023-03-30 03:57:40,742 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, ASSIGN 2023-03-30 03:57:40,743 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=false 2023-03-30 03:57:40,895 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=97a07dc2687acd6de72ca851295ed5b2, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:40,895 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148660895"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148660895"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148660895"}]},"ts":"1680148660895"} 2023-03-30 03:57:40,898 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:57:41,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 03:57:41,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:41,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 97a07dc2687acd6de72ca851295ed5b2, NAME => 'TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:57:41,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:57:41,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,063 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,065 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A 2023-03-30 03:57:41,065 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A 2023-03-30 03:57:41,066 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:41,066 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97a07dc2687acd6de72ca851295ed5b2 columnFamilyName A 2023-03-30 03:57:41,067 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(310): Store=97a07dc2687acd6de72ca851295ed5b2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:41,067 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,070 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B 2023-03-30 03:57:41,070 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B 2023-03-30 03:57:41,070 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:41,071 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97a07dc2687acd6de72ca851295ed5b2 columnFamilyName B 2023-03-30 03:57:41,072 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(310): Store=97a07dc2687acd6de72ca851295ed5b2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:41,073 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,075 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C 2023-03-30 03:57:41,075 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C 2023-03-30 03:57:41,075 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:41,075 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97a07dc2687acd6de72ca851295ed5b2 columnFamilyName C 2023-03-30 03:57:41,076 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(310): Store=97a07dc2687acd6de72ca851295ed5b2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:41,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:41,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:57:41,080 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:41,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:57:41,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 97a07dc2687acd6de72ca851295ed5b2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71636146, jitterRate=0.067461758852005}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:57:41,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:41,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., pid=37, masterSystemTime=1680148661052 2023-03-30 03:57:41,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:41,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:41,084 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=97a07dc2687acd6de72ca851295ed5b2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:41,085 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148661084"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148661084"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148661084"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148661084"}]},"ts":"1680148661084"} 2023-03-30 03:57:41,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-30 03:57:41,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 in 187 msec 2023-03-30 03:57:41,088 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-30 03:57:41,088 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, ASSIGN in 347 msec 2023-03-30 03:57:41,088 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 03:57:41,088 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148661088"}]},"ts":"1680148661088"} 2023-03-30 03:57:41,089 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 03:57:41,091 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 03:57:41,092 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1950 sec 2023-03-30 03:57:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 03:57:42,012 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-30 03:57:42,014 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x7889ece0 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:42,021 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@377ba302, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:42,023 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:42,025 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34534, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:42,026 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 03:57:42,027 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52472, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 03:57:42,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:57:42,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:57:42,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-30 03:57:42,441 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 03:57:42,443 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:57:42,443 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 03:57:42,446 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-30 03:57:42,450 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, REOPEN/MOVE}] 2023-03-30 03:57:42,451 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, REOPEN/MOVE 2023-03-30 03:57:42,452 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=97a07dc2687acd6de72ca851295ed5b2, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:42,452 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148662452"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148662452"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148662452"}]},"ts":"1680148662452"} 2023-03-30 03:57:42,453 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:57:42,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 97a07dc2687acd6de72ca851295ed5b2, disabling compactions & flushes 2023-03-30 03:57:42,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:42,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:42,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. after waiting 0 ms 2023-03-30 03:57:42,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:42,611 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-30 03:57:42,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:42,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:42,612 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 97a07dc2687acd6de72ca851295ed5b2 to self. 2023-03-30 03:57:42,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,614 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=97a07dc2687acd6de72ca851295ed5b2, regionState=CLOSED 2023-03-30 03:57:42,615 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148662614"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148662614"}]},"ts":"1680148662614"} 2023-03-30 03:57:42,618 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-30 03:57:42,619 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 in 163 msec 2023-03-30 03:57:42,619 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=true 2023-03-30 03:57:42,770 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=97a07dc2687acd6de72ca851295ed5b2, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:42,771 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148662770"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148662770"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148662770"}]},"ts":"1680148662770"} 2023-03-30 03:57:42,773 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:57:42,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:42,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 97a07dc2687acd6de72ca851295ed5b2, NAME => 'TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:57:42,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:57:42,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,941 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,942 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A 2023-03-30 03:57:42,943 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A 2023-03-30 03:57:42,943 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:42,943 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97a07dc2687acd6de72ca851295ed5b2 columnFamilyName A 2023-03-30 03:57:42,945 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(310): Store=97a07dc2687acd6de72ca851295ed5b2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:42,945 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,946 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B 2023-03-30 03:57:42,946 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B 2023-03-30 03:57:42,946 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:42,946 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97a07dc2687acd6de72ca851295ed5b2 columnFamilyName B 2023-03-30 03:57:42,947 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(310): Store=97a07dc2687acd6de72ca851295ed5b2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:42,947 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,948 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C 2023-03-30 03:57:42,948 DEBUG [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C 2023-03-30 03:57:42,948 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:57:42,948 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97a07dc2687acd6de72ca851295ed5b2 columnFamilyName C 2023-03-30 03:57:42,949 INFO [StoreOpener-97a07dc2687acd6de72ca851295ed5b2-1] regionserver.HStore(310): Store=97a07dc2687acd6de72ca851295ed5b2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:57:42,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:42,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:57:42,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:42,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 97a07dc2687acd6de72ca851295ed5b2; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64493914, jitterRate=-0.03896579146385193}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:57:42,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:42,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., pid=42, masterSystemTime=1680148662927 2023-03-30 03:57:42,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:42,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:42,958 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=97a07dc2687acd6de72ca851295ed5b2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:42,958 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148662958"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148662958"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148662958"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148662958"}]},"ts":"1680148662958"} 2023-03-30 03:57:42,961 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-30 03:57:42,961 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 in 186 msec 2023-03-30 03:57:42,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-30 03:57:42,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, REOPEN/MOVE in 511 msec 2023-03-30 03:57:42,965 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-30 03:57:42,965 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 517 msec 2023-03-30 03:57:42,967 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 938 msec 2023-03-30 03:57:42,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-30 03:57:42,970 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x29738198 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:42,977 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1eaa86ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:42,978 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x28a98302 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:42,991 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d63a8ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:42,992 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x39d5c003 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:42,997 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d80e5a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:42,998 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x39e3a826 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:43,003 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ea1a2c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:43,003 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x019e7fff to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:43,008 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25185572, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:43,009 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x35f0f398 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:43,024 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52249391, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:43,025 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x69fd9cc7 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:43,033 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ba3ce53, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:43,033 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x54da4907 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:43,048 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5144fcff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:43,049 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x52cea5eb to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:43,057 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54da4cc4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:43,058 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x3546dad6 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:57:43,066 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1280c068, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:57:43,076 DEBUG [hconnection-0x64ed056b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,077 DEBUG [hconnection-0x2f7b5370-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,077 DEBUG [hconnection-0x42511d64-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,077 DEBUG [hconnection-0x1677fdf8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,077 DEBUG [hconnection-0x6e1d00f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,077 DEBUG [hconnection-0x3d907a5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:43,078 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34538, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,078 DEBUG [hconnection-0x4182c38f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,079 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34562, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,079 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34608, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,082 DEBUG [hconnection-0x567bd078-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,083 DEBUG [hconnection-0x76be57cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,085 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34560, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,082 DEBUG [hconnection-0x300a9c3d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:57:43,085 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34550, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,085 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34592, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,088 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34580, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,088 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,090 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34612, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,090 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34610, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:57:43,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:43,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:43,095 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:43,095 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:43,097 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:43,097 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:43,100 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:43,100 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:43,104 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,104 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:43,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:43,104 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,104 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:43,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:43,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:43,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:43,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:43,107 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:43,107 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:43,107 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:43,107 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:43,107 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:43,107 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:43,107 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:43,108 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 03:57:43,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:43,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:43,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:43,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:43,111 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:43,111 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:43,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:43,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:43,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680148663122, max timestamp=1680148663132Num uniques 859; 2023-03-30 03:57:43,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680148663122, max timestamp=1680148663132Num uniques 859; 2023-03-30 03:57:43,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 03:57:43,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 03:57:43,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680148663122, max timestamp=1680148663132Num uniques 859; 2023-03-30 03:57:43,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 03:57:43,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148723140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148723140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148723140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148723141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148723141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148723242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148723242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148723242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148723242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148723242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148723449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148723449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148723449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148723449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148723450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,531 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330cb62cec2e3b14e5596ec5d9e0523ddec to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330cb62cec2e3b14e5596ec5d9e0523ddec 2023-03-30 03:57:43,535 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/45036a4afec4439780c175835ca6dbab 2023-03-30 03:57:43,547 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/a7786a90333f4a51aefa6c9f347f8d6b 2023-03-30 03:57:43,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148723753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148723753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148723753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148723755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148723755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,955 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/dad9d3273390487d9147ec8bb3d71a1e 2023-03-30 03:57:43,958 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/45036a4afec4439780c175835ca6dbab as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/45036a4afec4439780c175835ca6dbab 2023-03-30 03:57:43,966 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/45036a4afec4439780c175835ca6dbab, entries=150, sequenceid=13, filesize=25.1 K 2023-03-30 03:57:43,967 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/a7786a90333f4a51aefa6c9f347f8d6b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/a7786a90333f4a51aefa6c9f347f8d6b 2023-03-30 03:57:43,970 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/a7786a90333f4a51aefa6c9f347f8d6b, entries=150, sequenceid=13, filesize=11.6 K 2023-03-30 03:57:43,971 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/dad9d3273390487d9147ec8bb3d71a1e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dad9d3273390487d9147ec8bb3d71a1e 2023-03-30 03:57:43,973 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dad9d3273390487d9147ec8bb3d71a1e, entries=150, sequenceid=13, filesize=11.6 K 2023-03-30 03:57:43,974 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=154.31 KB/158010 for 97a07dc2687acd6de72ca851295ed5b2 in 866ms, sequenceid=13, compaction requested=false 2023-03-30 03:57:43,974 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-30 03:57:43,975 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:43,975 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:43,975 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:43,975 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:43,975 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,975 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:43,975 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:43,976 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,976 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:43,976 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:43,976 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:43,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:43,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b97d8a1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:43,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:43,978 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:43,978 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:43,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:43,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:43,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:43,978 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,978 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:43,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:43,979 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:43,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:43,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:43,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:43,981 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:43,981 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:43,981 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:43,981 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:43,981 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:43,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:43,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:43,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:43,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:43,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:43,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:43,983 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:43,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:43,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:43,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:43,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:43,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:43,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:43,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,985 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:43,985 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:43,985 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:43,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:43,985 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:43,985 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:43,985 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:43,985 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:43,985 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,085 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:44,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:44,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:44,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:44,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:44,088 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:44,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:44,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:44,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:44,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:44,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,088 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:44,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:44,089 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,089 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:44,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:44,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:44,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:44,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:44,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:44,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:44,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:44,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:44,090 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:44,090 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:44,090 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:44,090 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-03-30 03:57:44,090 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:44,090 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:44,090 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:44,090 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:44,090 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:44,090 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:44,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:44,264 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:44,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148724265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148724265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148724266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148724266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148724267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148724372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148724372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148724376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148724376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148724376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,501 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330eb45f9ce9e174a61941d50e81c4118aa to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330eb45f9ce9e174a61941d50e81c4118aa 2023-03-30 03:57:44,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148724576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148724576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148724583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148724583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148724583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,743 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 03:57:44,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148724882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148724882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148724888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148724888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:44,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148724890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,905 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/2f285c3b4f504ef5ad4811ba30276b89 2023-03-30 03:57:44,918 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/8b726bf3fbbf498b88659e955762c333 2023-03-30 03:57:44,937 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/3b2fc136bfd243eeb3f2f12f35fafeec 2023-03-30 03:57:44,940 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/2f285c3b4f504ef5ad4811ba30276b89 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/2f285c3b4f504ef5ad4811ba30276b89 2023-03-30 03:57:44,943 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/2f285c3b4f504ef5ad4811ba30276b89, entries=150, sequenceid=49, filesize=25.1 K 2023-03-30 03:57:44,944 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/8b726bf3fbbf498b88659e955762c333 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/8b726bf3fbbf498b88659e955762c333 2023-03-30 03:57:44,947 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/8b726bf3fbbf498b88659e955762c333, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 03:57:44,947 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/3b2fc136bfd243eeb3f2f12f35fafeec as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3b2fc136bfd243eeb3f2f12f35fafeec 2023-03-30 03:57:44,951 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3b2fc136bfd243eeb3f2f12f35fafeec, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 03:57:44,952 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=33.54 KB/34350 for 97a07dc2687acd6de72ca851295ed5b2 in 862ms, sequenceid=49, compaction requested=false 2023-03-30 03:57:44,952 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:44,952 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:44,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:44,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:44,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:44,952 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:44,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,953 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:44,954 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:44,954 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:44,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:44,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5efbbf19[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:44,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:44,955 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:44,956 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:44,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:44,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:44,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:44,956 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,956 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:44,956 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:44,957 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:44,957 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,957 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:44,957 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:44,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:44,959 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:44,959 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:44,959 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:44,959 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:44,959 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:44,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:44,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:44,961 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:44,962 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,964 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:44,964 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:44,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:44,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:44,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:44,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:44,964 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:44,964 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:44,965 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,965 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:44,965 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:44,965 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:44,965 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:44,965 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:44,965 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:44,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:44,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:44,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:45,064 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:45,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:45,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:45,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:45,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:45,070 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:45,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:45,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:45,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:45,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:45,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:45,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:45,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:45,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:45,071 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:45,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:45,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:45,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:45,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:45,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:45,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:45,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:45,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:45,086 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:45,086 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:45,087 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 03:57:45,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:45,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:45,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:45,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:45,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:45,087 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:45,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:45,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148665458, max timestamp=1680148665475Num uniques 859; 2023-03-30 03:57:45,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148665458, max timestamp=1680148665475Num uniques 859; 2023-03-30 03:57:45,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:45,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148665458, max timestamp=1680148665475Num uniques 859; 2023-03-30 03:57:45,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:45,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:45,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148725512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148725515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148725520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148725521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148725524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,559 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303302df676b9b2ab4897b14bbe444c8cb316 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302df676b9b2ab4897b14bbe444c8cb316 2023-03-30 03:57:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148725625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148725628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148725636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148725636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148725637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148725832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148725841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148725844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148725844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:45,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:45,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148725845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,004 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=57, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/e97f33a262f241acbc1d92d04d7f02a2 2023-03-30 03:57:46,048 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/727163f3aa2a43c6bd8592f0965214f7 2023-03-30 03:57:46,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:46,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148726141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:46,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148726152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:46,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148726152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:46,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148726154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:46,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148726154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,505 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/2f6dba1dab9249008ae34a273e11a705 2023-03-30 03:57:46,509 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/e97f33a262f241acbc1d92d04d7f02a2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e97f33a262f241acbc1d92d04d7f02a2 2023-03-30 03:57:46,513 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e97f33a262f241acbc1d92d04d7f02a2, entries=100, sequenceid=57, filesize=18.4 K 2023-03-30 03:57:46,514 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/727163f3aa2a43c6bd8592f0965214f7 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/727163f3aa2a43c6bd8592f0965214f7 2023-03-30 03:57:46,518 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/727163f3aa2a43c6bd8592f0965214f7, entries=100, sequenceid=57, filesize=9.3 K 2023-03-30 03:57:46,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/2f6dba1dab9249008ae34a273e11a705 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/2f6dba1dab9249008ae34a273e11a705 2023-03-30 03:57:46,523 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/2f6dba1dab9249008ae34a273e11a705, entries=100, sequenceid=57, filesize=9.3 K 2023-03-30 03:57:46,524 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 97a07dc2687acd6de72ca851295ed5b2 in 1438ms, sequenceid=57, compaction requested=true 2023-03-30 03:57:46,524 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:46,524 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:46,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:46,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:46,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:46,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:46,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:46,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55cbc1cc[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:46,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:46,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,528 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:46,528 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:46,528 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:46,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:46,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:46,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:46,529 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:46,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:46,531 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:46,531 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:46,531 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:46,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:46,532 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:46,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,537 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,538 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,540 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,541 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:46,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:46,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:46,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,542 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:46,542 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,542 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:46,542 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:46,543 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:46,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:46,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:46,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:46,643 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:46,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:46,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:46,659 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:46,659 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:46,659 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:46,659 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:46,660 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,660 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:46,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680148665512, max timestamp=1680148666658Num uniques 1118; 2023-03-30 03:57:46,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-30 03:57:46,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680148665512, max timestamp=1680148666658Num uniques 1118; 2023-03-30 03:57:46,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-30 03:57:46,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680148665512, max timestamp=1680148666658Num uniques 1118; 2023-03-30 03:57:46,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-30 03:57:46,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=26.84 KB heapSize=65.39 KB 2023-03-30 03:57:46,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:46,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:46,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:46,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:46,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:46,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:46,663 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:46,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:46,664 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,664 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,664 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:46,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:46,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:46,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:46,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:46,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:46,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:46,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:46,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:46,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:46,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:46,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:46,679 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:46,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:46,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f786d1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:46,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:46,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,684 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,684 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,684 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:46,684 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:46,684 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:46,684 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:46,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:46,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:46,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:46,688 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:46,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:46,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:46,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:46,688 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:46,689 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,689 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,689 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:46,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:46,690 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,691 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:46,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:46,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:46,693 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:46,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:46,694 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:46,694 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:46,696 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330c188f6828511487daf508da0280053d6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c188f6828511487daf508da0280053d6 2023-03-30 03:57:46,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:46,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148666719, max timestamp=1680148666732Num uniques 859; 2023-03-30 03:57:46,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148666719, max timestamp=1680148666732Num uniques 859; 2023-03-30 03:57:46,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:46,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:46,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148666719, max timestamp=1680148666732Num uniques 859; 2023-03-30 03:57:46,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:57:46,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:46,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680148666764, max timestamp=1680148666780Num uniques 1118; 2023-03-30 03:57:46,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680148666764, max timestamp=1680148666780Num uniques 1118; 2023-03-30 03:57:46,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-30 03:57:46,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148726773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-30 03:57:46,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680148666764, max timestamp=1680148666780Num uniques 1118; 2023-03-30 03:57:46,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-30 03:57:46,794 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:46,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:46,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:46,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:46,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:46,803 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:46,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:46,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:46,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:46,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:46,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:46,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:46,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:46,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:46,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:46,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:46,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:46,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:46,807 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:46,807 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:46,807 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:46,807 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:46,808 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,808 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,809 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,809 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:46,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@230065e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:46,810 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:46,810 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,810 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,811 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:46,811 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:46,811 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:46,811 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,811 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:46,811 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:46,811 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:46,811 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,812 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:46,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:46,813 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:46,813 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:46,813 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:46,813 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:46,813 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:46,814 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,814 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:46,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,814 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:46,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:46,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:46,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:46,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:46,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:46,837 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:46,837 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:46,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680148666797, max timestamp=1680148666842Num uniques 1127; 2023-03-30 03:57:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-30 03:57:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680148666797, max timestamp=1680148666842Num uniques 1127; 2023-03-30 03:57:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-30 03:57:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680148666797, max timestamp=1680148666842Num uniques 1127; 2023-03-30 03:57:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-30 03:57:46,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680148666871, max timestamp=1680148666894Num uniques 1136; 2023-03-30 03:57:46,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-30 03:57:46,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680148666871, max timestamp=1680148666894Num uniques 1136; 2023-03-30 03:57:46,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680148666871, max timestamp=1680148666894Num uniques 1136; 2023-03-30 03:57:46,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-30 03:57:46,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-03-30 03:57:46,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680148666916, max timestamp=1680148666924Num uniques 1145; 2023-03-30 03:57:46,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-30 03:57:46,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680148666916, max timestamp=1680148666924Num uniques 1145; 2023-03-30 03:57:46,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-30 03:57:46,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680148666916, max timestamp=1680148666924Num uniques 1145; 2023-03-30 03:57:46,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-03-30 03:57:46,937 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:46,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:46,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:46,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:46,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:46,940 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:46,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:46,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:46,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:46,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,941 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,941 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:46,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:46,941 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,941 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:46,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:46,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:46,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:46,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:46,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:46,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:46,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:46,942 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:46,943 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:46,943 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:46,943 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:46,943 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:46,943 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:46,943 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,943 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:46,943 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:46,943 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,943 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,944 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,944 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:46,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1306276[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:46,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:46,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:46,946 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,946 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:46,946 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,946 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:46,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:46,948 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:46,948 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:46,948 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:46,948 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:46,948 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:46,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,949 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:46,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,950 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:46,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,952 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:46,952 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:46,952 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:46,953 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:46,952 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:46,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:46,953 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:46,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:46,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:46,953 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:46,953 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:46,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:46,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:46,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:46,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148726952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:46,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148726953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:46,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680148666948, max timestamp=1680148666962Num uniques 1154; 2023-03-30 03:57:46,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-30 03:57:46,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680148666948, max timestamp=1680148666962Num uniques 1154; 2023-03-30 03:57:46,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-30 03:57:46,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680148666948, max timestamp=1680148666962Num uniques 1154; 2023-03-30 03:57:46,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-30 03:57:47,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148727031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680148667024, max timestamp=1680148667032Num uniques 1113; 2023-03-30 03:57:47,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-03-30 03:57:47,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680148667024, max timestamp=1680148667032Num uniques 1113; 2023-03-30 03:57:47,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-03-30 03:57:47,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680148667024, max timestamp=1680148667032Num uniques 1113; 2023-03-30 03:57:47,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-03-30 03:57:47,053 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:47,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:47,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:47,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:47,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:47,056 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:47,056 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,056 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:47,056 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:47,057 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,057 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,058 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:47,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:47,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:47,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:47,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:47,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:47,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:47,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:47,060 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,060 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:47,060 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:47,060 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:47,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:47,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:47,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:47,061 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,061 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:47,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@afe163d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:47,062 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:47,062 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,063 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,063 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,063 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:47,064 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:47,064 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:47,064 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:47,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:47,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:47,069 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:47,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:47,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:47,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:47,070 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:47,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,071 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:47,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:47,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,074 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,074 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:47,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,074 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:47,075 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,075 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1680148667067, max timestamp=1680148667080Num uniques 1022; 2023-03-30 03:57:47,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=14742, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-03-30 03:57:47,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1680148667067, max timestamp=1680148667080Num uniques 1022; 2023-03-30 03:57:47,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=14742, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-03-30 03:57:47,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1680148667067, max timestamp=1680148667080Num uniques 1022; 2023-03-30 03:57:47,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=14742, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-03-30 03:57:47,101 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/740656a63ab047f28b244f8f9c508ef3 2023-03-30 03:57:47,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148727117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680148667105, max timestamp=1680148667127Num uniques 1181; 2023-03-30 03:57:47,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=10576, suffix heap size=150792,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=231 2023-03-30 03:57:47,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680148667105, max timestamp=1680148667127Num uniques 1181; 2023-03-30 03:57:47,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=10576, suffix heap size=150792,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=231 2023-03-30 03:57:47,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680148667105, max timestamp=1680148667127Num uniques 1181; 2023-03-30 03:57:47,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=10576, suffix heap size=150792,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=231 2023-03-30 03:57:47,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680148667156, max timestamp=1680148667164Num uniques 1090; 2023-03-30 03:57:47,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-30 03:57:47,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680148667156, max timestamp=1680148667164Num uniques 1090; 2023-03-30 03:57:47,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-30 03:57:47,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680148667156, max timestamp=1680148667164Num uniques 1090; 2023-03-30 03:57:47,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-30 03:57:47,174 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:47,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:47,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:47,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:47,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:47,177 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:47,177 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,177 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:47,177 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:47,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,188 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,188 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,188 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,188 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:47,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:47,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:47,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:47,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:47,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:47,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:47,189 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,190 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:47,190 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:47,190 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:47,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:47,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:47,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:47,191 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,191 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,191 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,191 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:47,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:47,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2194c8bd[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:47,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,193 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:47,193 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:47,193 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:47,193 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:47,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:47,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:47,199 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:47,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:47,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:47,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:47,199 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:47,201 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,201 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,202 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,203 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,204 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,204 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,205 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:47,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:47,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,205 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:47,205 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:47,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:47,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680148667215, max timestamp=1680148667244Num uniques 1149; 2023-03-30 03:57:47,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-30 03:57:47,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680148667215, max timestamp=1680148667244Num uniques 1149; 2023-03-30 03:57:47,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-30 03:57:47,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680148667215, max timestamp=1680148667244Num uniques 1149; 2023-03-30 03:57:47,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-30 03:57:47,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680148667273, max timestamp=1680148667285Num uniques 1108; 2023-03-30 03:57:47,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680148667273, max timestamp=1680148667285Num uniques 1108; 2023-03-30 03:57:47,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-30 03:57:47,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680148667273, max timestamp=1680148667285Num uniques 1108; 2023-03-30 03:57:47,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-30 03:57:47,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-30 03:57:47,305 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:47,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:47,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:47,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:47,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:47,308 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:47,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:47,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:47,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,310 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:47,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:47,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:47,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:47,312 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,312 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:47,312 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:47,312 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,312 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:47,312 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:47,312 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,312 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:47,312 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:47,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,313 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,313 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,314 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:47,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23943827[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:47,315 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:47,315 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,316 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,317 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:47,317 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:47,317 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:47,317 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:47,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:47,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,318 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:47,318 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:47,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:47,321 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:47,321 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:47,321 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:47,321 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:47,321 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:47,321 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,322 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,322 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,323 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,324 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,324 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:47,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:47,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,325 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:47,325 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:47,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:47,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680148667301, max timestamp=1680148667327Num uniques 1067; 2023-03-30 03:57:47,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-30 03:57:47,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680148667301, max timestamp=1680148667327Num uniques 1067; 2023-03-30 03:57:47,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680148667301, max timestamp=1680148667327Num uniques 1067; 2023-03-30 03:57:47,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-30 03:57:47,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-30 03:57:47,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680148667353, max timestamp=1680148667363Num uniques 1026; 2023-03-30 03:57:47,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-30 03:57:47,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680148667353, max timestamp=1680148667363Num uniques 1026; 2023-03-30 03:57:47,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-30 03:57:47,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680148667353, max timestamp=1680148667363Num uniques 1026; 2023-03-30 03:57:47,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-30 03:57:47,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=385, cellSize=17630, totalHeapSize=40280, min timestamp=1680148667372, max timestamp=1680148667397Num uniques 1135; 2023-03-30 03:57:47,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=17630, suffix heap size=146008,new segment heap size=40280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=385 2023-03-30 03:57:47,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=385, cellSize=17630, totalHeapSize=40280, min timestamp=1680148667372, max timestamp=1680148667397Num uniques 1135; 2023-03-30 03:57:47,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=17630, suffix heap size=146008,new segment heap size=40280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=385 2023-03-30 03:57:47,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680148667387, max timestamp=1680148667397Num uniques 1135; 2023-03-30 03:57:47,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-30 03:57:47,425 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:47,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:47,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:47,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:47,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:47,434 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:47,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:47,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:47,435 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,435 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,435 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,436 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680148667419, max timestamp=1680148667433Num uniques 1244; 2023-03-30 03:57:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=56974, new segment data size=13464, suffix heap size=157344,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1244, new segment cells count=294 2023-03-30 03:57:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680148667419, max timestamp=1680148667433Num uniques 1094; 2023-03-30 03:57:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680148667419, max timestamp=1680148667433Num uniques 1244; 2023-03-30 03:57:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-30 03:57:47,436 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,436 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=56974, new segment data size=13464, suffix heap size=157344,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1244, new segment cells count=294 2023-03-30 03:57:47,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:47,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:47,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:47,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:47,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:47,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:47,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:47,438 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,438 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:47,438 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:47,438 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:47,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:47,439 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,439 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:47,439 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:47,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:47,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21b5cdf1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:47,442 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:47,442 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,442 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,443 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:47,443 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:47,443 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:47,443 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,443 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:47,444 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,444 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:47,444 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,444 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:47,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:47,446 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:47,446 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:47,446 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:47,446 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:47,446 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:47,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,448 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,449 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,450 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:47,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,451 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,450 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,451 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:47,451 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:47,451 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,451 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,451 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:47,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148727466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680148667463, max timestamp=1680148667478Num uniques 1153; 2023-03-30 03:57:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-03-30 03:57:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680148667463, max timestamp=1680148667478Num uniques 1153; 2023-03-30 03:57:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680148667463, max timestamp=1680148667478Num uniques 1153; 2023-03-30 03:57:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-03-30 03:57:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=11585, suffix heap size=147880,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=253 2023-03-30 03:57:47,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/6d41ce0d315b49089131960fa5987ce3 2023-03-30 03:57:47,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/d66ae52f7ee2454a80533ff8f0413a09 2023-03-30 03:57:47,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/740656a63ab047f28b244f8f9c508ef3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/740656a63ab047f28b244f8f9c508ef3 2023-03-30 03:57:47,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/740656a63ab047f28b244f8f9c508ef3, entries=200, sequenceid=95, filesize=31.9 K 2023-03-30 03:57:47,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/6d41ce0d315b49089131960fa5987ce3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/6d41ce0d315b49089131960fa5987ce3 2023-03-30 03:57:47,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680148667508, max timestamp=1680148667523Num uniques 1112; 2023-03-30 03:57:47,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-03-30 03:57:47,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680148667508, max timestamp=1680148667523Num uniques 1112; 2023-03-30 03:57:47,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-03-30 03:57:47,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1680148667508, max timestamp=1680148667523Num uniques 1112; 2023-03-30 03:57:47,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-03-30 03:57:47,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/6d41ce0d315b49089131960fa5987ce3, entries=150, sequenceid=95, filesize=11.6 K 2023-03-30 03:57:47,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/d66ae52f7ee2454a80533ff8f0413a09 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d66ae52f7ee2454a80533ff8f0413a09 2023-03-30 03:57:47,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d66ae52f7ee2454a80533ff8f0413a09, entries=150, sequenceid=95, filesize=11.6 K 2023-03-30 03:57:47,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=33.54 KB/34350 for 97a07dc2687acd6de72ca851295ed5b2 in 870ms, sequenceid=95, compaction requested=true 2023-03-30 03:57:47,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:47,532 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:47,532 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:47,533 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:47,533 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/B is initiating minor compaction (all files) 2023-03-30 03:57:47,533 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/B in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,533 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/a7786a90333f4a51aefa6c9f347f8d6b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/8b726bf3fbbf498b88659e955762c333, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/727163f3aa2a43c6bd8592f0965214f7, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/6d41ce0d315b49089131960fa5987ce3] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=44.1 K 2023-03-30 03:57:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:47,533 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 102976 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:47,534 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/A is initiating minor compaction (all files) 2023-03-30 03:57:47,534 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/A in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,534 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/45036a4afec4439780c175835ca6dbab, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/2f285c3b4f504ef5ad4811ba30276b89, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e97f33a262f241acbc1d92d04d7f02a2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/740656a63ab047f28b244f8f9c508ef3] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=100.6 K 2023-03-30 03:57:47,534 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a7786a90333f4a51aefa6c9f347f8d6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680148663100 2023-03-30 03:57:47,534 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8b726bf3fbbf498b88659e955762c333, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148663135 2023-03-30 03:57:47,534 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 45036a4afec4439780c175835ca6dbab, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680148663100 2023-03-30 03:57:47,534 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 727163f3aa2a43c6bd8592f0965214f7, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680148664265 2023-03-30 03:57:47,535 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=60.38 KB heapSize=160.88 KB 2023-03-30 03:57:47,535 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6d41ce0d315b49089131960fa5987ce3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680148665512 2023-03-30 03:57:47,536 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 2f285c3b4f504ef5ad4811ba30276b89, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148663135 2023-03-30 03:57:47,536 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting e97f33a262f241acbc1d92d04d7f02a2, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680148664265 2023-03-30 03:57:47,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:47,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:47,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:47,536 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 740656a63ab047f28b244f8f9c508ef3, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680148665512 2023-03-30 03:57:47,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:47,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:47,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:47,547 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#B#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:47,551 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:47,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:47,561 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#A#compaction#320 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:47,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:47,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:47,562 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:47,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:47,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:47,563 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,563 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,564 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,564 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,564 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,564 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:47,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:47,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:47,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:47,588 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:47,588 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,588 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:47,589 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:47,589 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:47,589 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,589 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:47,589 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:47,589 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,589 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:47,589 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:47,589 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,589 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:47,589 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:47,590 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,591 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,593 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033081df6b1f925d48d287e93ebee05bfc61 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033081df6b1f925d48d287e93ebee05bfc61 2023-03-30 03:57:47,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:47,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b807804[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:47,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:47,593 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,594 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:47,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:47,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:47,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,595 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:47,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:47,597 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:47,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:47,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:47,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:47,598 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:47,599 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,599 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,599 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:47,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:47,600 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,606 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,606 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,606 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:47,606 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,606 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:47,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:47,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:47,606 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,607 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:47,607 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:47,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:47,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:47,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148727575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148727581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,621 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=413, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/667fbfbf499844fd881066672e98de3f 2023-03-30 03:57:47,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148727613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148727613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148727613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/e021c2d717e042aeb52c0e52ead164a2 2023-03-30 03:57:47,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/a19a7ac2261d4cfaa9766ec6134592e8 2023-03-30 03:57:47,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/667fbfbf499844fd881066672e98de3f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/667fbfbf499844fd881066672e98de3f 2023-03-30 03:57:47,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/667fbfbf499844fd881066672e98de3f, entries=300, sequenceid=413, filesize=46.1 K 2023-03-30 03:57:47,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/e021c2d717e042aeb52c0e52ead164a2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/e021c2d717e042aeb52c0e52ead164a2 2023-03-30 03:57:47,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/e021c2d717e042aeb52c0e52ead164a2, entries=150, sequenceid=413, filesize=11.9 K 2023-03-30 03:57:47,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/a19a7ac2261d4cfaa9766ec6134592e8 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a19a7ac2261d4cfaa9766ec6134592e8 2023-03-30 03:57:47,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a19a7ac2261d4cfaa9766ec6134592e8, entries=150, sequenceid=413, filesize=11.9 K 2023-03-30 03:57:47,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~180.75 KB/185088, currentSize=114.05 KB/116790 for 97a07dc2687acd6de72ca851295ed5b2 in 152ms, sequenceid=413, compaction requested=true 2023-03-30 03:57:47,687 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:47,687 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-30 03:57:47,687 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-30 03:57:47,687 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-30 03:57:47,707 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:47,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:47,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:47,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:47,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:47,709 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:47,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:47,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:47,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,710 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:47,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:47,711 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,711 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:47,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:47,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:47,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:47,712 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:47,712 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:47,712 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:47,712 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,712 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:47,712 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:57:47,712 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:47,712 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:47,712 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:47,712 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:47,712 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:47,712 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:47,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:47,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:47,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148727749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148727750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148727750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148727751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148727814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148727857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148727857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148727857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:47,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148727858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:47,959 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/c98caaf090194ebb8d65859777989d4c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c98caaf090194ebb8d65859777989d4c 2023-03-30 03:57:47,962 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/B of 97a07dc2687acd6de72ca851295ed5b2 into c98caaf090194ebb8d65859777989d4c(size=11.7 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:47,962 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:47,962 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/B, priority=12, startTime=1680148667532; duration=0sec 2023-03-30 03:57:47,962 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-30 03:57:47,962 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 4 compacting, 1 eligible, 16 blocking 2023-03-30 03:57:47,963 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-30 03:57:47,963 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-30 03:57:47,963 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. because compaction request was cancelled 2023-03-30 03:57:47,963 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:57:47,964 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:57:47,964 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/C is initiating minor compaction (all files) 2023-03-30 03:57:47,965 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/C in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:47,965 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dad9d3273390487d9147ec8bb3d71a1e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3b2fc136bfd243eeb3f2f12f35fafeec, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/2f6dba1dab9249008ae34a273e11a705, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d66ae52f7ee2454a80533ff8f0413a09, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a19a7ac2261d4cfaa9766ec6134592e8] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=56.0 K 2023-03-30 03:57:47,965 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting dad9d3273390487d9147ec8bb3d71a1e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680148663100 2023-03-30 03:57:47,965 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3b2fc136bfd243eeb3f2f12f35fafeec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680148663135 2023-03-30 03:57:47,966 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2f6dba1dab9249008ae34a273e11a705, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680148664265 2023-03-30 03:57:47,966 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d66ae52f7ee2454a80533ff8f0413a09, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680148665512 2023-03-30 03:57:47,967 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a19a7ac2261d4cfaa9766ec6134592e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680148667532 2023-03-30 03:57:47,974 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#C#compaction#324 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:48,046 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/145e81aebdad47a0958e6beb99ef3e96 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/145e81aebdad47a0958e6beb99ef3e96 2023-03-30 03:57:48,050 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/A of 97a07dc2687acd6de72ca851295ed5b2 into 145e81aebdad47a0958e6beb99ef3e96(size=25.3 K), total size for store is 71.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:48,050 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:48,050 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/A, priority=12, startTime=1680148667532; duration=0sec 2023-03-30 03:57:48,050 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-30 03:57:48,050 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-03-30 03:57:48,050 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-30 03:57:48,051 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-30 03:57:48,051 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. because compaction request was cancelled 2023-03-30 03:57:48,051 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-30 03:57:48,051 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-30 03:57:48,051 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-30 03:57:48,051 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. because compaction request was cancelled 2023-03-30 03:57:48,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148728069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148728071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148728074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148728076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,126 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303303e38196fd87e414c8428cdb290edc4a3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303e38196fd87e414c8428cdb290edc4a3 2023-03-30 03:57:48,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34560 deadline: 1680148728121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,136 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=433, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/bbb75f81063f4a6086b1c1cd0b84666b 2023-03-30 03:57:48,160 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/c27264af13dd43aebe786aa6e43cb730 2023-03-30 03:57:48,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148728371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148728378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148728379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148728379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,390 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/c02679990ca442ad853b54a2a7b25d0b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c02679990ca442ad853b54a2a7b25d0b 2023-03-30 03:57:48,394 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/C of 97a07dc2687acd6de72ca851295ed5b2 into c02679990ca442ad853b54a2a7b25d0b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:48,394 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:48,394 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/C, priority=11, startTime=1680148667687; duration=0sec 2023-03-30 03:57:48,394 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:48,581 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/d7091aa6f9ee4c03ba75cf5d9910ef99 2023-03-30 03:57:48,586 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/bbb75f81063f4a6086b1c1cd0b84666b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bbb75f81063f4a6086b1c1cd0b84666b 2023-03-30 03:57:48,591 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bbb75f81063f4a6086b1c1cd0b84666b, entries=150, sequenceid=433, filesize=25.4 K 2023-03-30 03:57:48,592 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/c27264af13dd43aebe786aa6e43cb730 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c27264af13dd43aebe786aa6e43cb730 2023-03-30 03:57:48,597 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c27264af13dd43aebe786aa6e43cb730, entries=150, sequenceid=433, filesize=11.9 K 2023-03-30 03:57:48,597 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/d7091aa6f9ee4c03ba75cf5d9910ef99 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d7091aa6f9ee4c03ba75cf5d9910ef99 2023-03-30 03:57:48,602 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d7091aa6f9ee4c03ba75cf5d9910ef99, entries=150, sequenceid=433, filesize=11.9 K 2023-03-30 03:57:48,602 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 97a07dc2687acd6de72ca851295ed5b2 in 890ms, sequenceid=433, compaction requested=true 2023-03-30 03:57:48,603 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:48,603 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:48,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:48,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:48,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:48,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:48,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:48,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cd682e9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:48,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:48,606 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,606 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:48,608 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,608 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:48,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:48,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:48,609 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:48,609 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:48,612 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:48,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:48,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:48,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:48,612 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:48,623 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,623 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:48,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,625 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,626 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:48,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:48,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:48,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:48,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:48,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:48,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,632 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,632 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,632 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,632 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:48,633 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:48,633 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:48,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:48,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:57:48,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:48,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:48,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:48,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:48,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:48,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:48,661 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330f4aeba16768441a5b758693189c8f850 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f4aeba16768441a5b758693189c8f850 2023-03-30 03:57:48,733 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:48,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:48,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:48,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:48,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:48,738 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:48,738 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:48,738 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:48,738 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:48,741 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,741 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,742 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:48,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:48,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:48,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:48,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:48,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:48,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:48,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:48,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:48,762 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:48,766 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:48,766 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:48,766 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:48,766 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:48,767 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,767 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,767 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,768 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e01cc54[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:48,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:48,770 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,771 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:48,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:48,771 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:48,771 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,771 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148728768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,779 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,779 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:48,779 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:48,779 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:48,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:48,782 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:48,782 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:48,782 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:48,782 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:48,782 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:48,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,841 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:48,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,841 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,842 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:48,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:48,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:48,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:48,844 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:48,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:48,844 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:48,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148728880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34580 deadline: 1680148728884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34562 deadline: 1680148728891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148728891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148728893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,944 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:48,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:48,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:48,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:48,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:48,949 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:48,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:48,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:48,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:48,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,950 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:48,950 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:48,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:48,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:48,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:48,958 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:48,958 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:48,958 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:48,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:48,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:48,961 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:48,961 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:48,961 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:48,961 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:48,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:48,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:48,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:48,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:48,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,963 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:48,965 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:48,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51545820[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:48,965 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,965 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:48,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:48,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:48,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:48,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:48,966 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,967 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,967 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:48,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:48,970 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:48,970 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:48,970 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:48,970 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:48,970 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:48,971 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:48,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:48,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:48,973 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:48,975 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:48,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:48,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:48,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:48,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:48,977 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:48,977 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:48,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:48,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:48,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:49,067 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=449, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/490aa8e9d99241e9988760b384e72cec 2023-03-30 03:57:49,077 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:49,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:49,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:49,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:49,094 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:49,094 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,094 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:49,094 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:49,095 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,095 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,095 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,095 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,096 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,096 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:49,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:49,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148729090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:49,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:49,114 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:49,114 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,114 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:49,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:49,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:49,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:49,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:49,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:49,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:49,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:49,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:49,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:49,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:49,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@94c5a56[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:49,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:49,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,131 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:49,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:49,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:49,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:49,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:49,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:49,136 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:49,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:49,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:49,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:49,136 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:49,137 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,137 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:49,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,137 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,139 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/304114c6000c47bfb3a64eb529912be2 2023-03-30 03:57:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,141 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,148 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:49,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:49,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:49,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,149 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:49,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,149 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,149 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:49,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:49,249 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:49,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:49,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:49,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:49,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:49,266 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:49,266 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,266 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:49,266 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:49,266 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,266 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,268 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,268 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,268 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,268 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:49,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:49,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:49,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:49,270 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:49,270 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,270 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:49,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:49,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:49,274 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,274 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:49,274 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:49,274 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:49,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:49,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:49,274 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:49,280 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,280 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:49,283 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61e5405c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:49,283 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,289 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:49,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:49,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:49,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:49,298 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,298 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,298 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:49,298 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:49,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:49,300 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:49,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:49,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:49,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:49,300 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:49,307 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,308 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,308 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:49,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,308 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:49,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,314 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:49,314 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:49,314 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:49,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:49,314 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:49,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:49,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148729400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,414 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:49,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:49,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:49,418 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:49,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:49,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:49,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,418 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,419 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,419 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,421 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:49,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:49,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:49,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:49,429 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:49,429 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,429 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:49,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:49,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:49,430 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,430 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:49,430 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:49,430 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:49,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:49,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:49,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:49,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,440 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:49,442 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:49,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31d65358[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:49,442 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,454 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,454 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:49,454 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:49,455 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:49,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:49,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,455 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,455 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:49,456 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:49,459 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:49,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:49,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:49,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:49,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:49,460 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:49,460 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,460 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,461 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:49,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,462 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,463 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:49,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:49,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,465 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:49,465 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:49,465 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,465 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:49,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/3704da99fa5049c0bb1104d5eb2bc32e 2023-03-30 03:57:49,566 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:49,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/490aa8e9d99241e9988760b384e72cec as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/490aa8e9d99241e9988760b384e72cec 2023-03-30 03:57:49,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:49,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:49,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:49,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:49,570 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:49,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:49,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:49,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,571 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,571 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/490aa8e9d99241e9988760b384e72cec, entries=200, sequenceid=449, filesize=32.3 K 2023-03-30 03:57:49,571 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,571 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/304114c6000c47bfb3a64eb529912be2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/304114c6000c47bfb3a64eb529912be2 2023-03-30 03:57:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:49,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:49,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:49,572 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:49,572 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,573 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:49,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/304114c6000c47bfb3a64eb529912be2, entries=150, sequenceid=449, filesize=11.9 K 2023-03-30 03:57:49,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:49,578 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:49,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/3704da99fa5049c0bb1104d5eb2bc32e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3704da99fa5049c0bb1104d5eb2bc32e 2023-03-30 03:57:49,578 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:49,578 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:49,578 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:49,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:49,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:49,578 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:49,579 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,579 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,580 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:49,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4230881a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:49,581 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:49,581 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3704da99fa5049c0bb1104d5eb2bc32e, entries=150, sequenceid=449, filesize=11.9 K 2023-03-30 03:57:49,581 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,581 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,581 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:49,582 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:49,581 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:49,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 97a07dc2687acd6de72ca851295ed5b2 in 946ms, sequenceid=449, compaction requested=true 2023-03-30 03:57:49,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:49,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:49,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:49,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:49,582 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:49,582 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:49,582 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:57:49,584 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48582 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:49,584 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/B is initiating minor compaction (all files) 2023-03-30 03:57:49,584 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,584 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,585 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 132186 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:57:49,584 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:49,585 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/A is initiating minor compaction (all files) 2023-03-30 03:57:49,584 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/B in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,585 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/A in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,585 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c98caaf090194ebb8d65859777989d4c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/e021c2d717e042aeb52c0e52ead164a2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c27264af13dd43aebe786aa6e43cb730, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/304114c6000c47bfb3a64eb529912be2] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=47.4 K 2023-03-30 03:57:49,585 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c98caaf090194ebb8d65859777989d4c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680148665512 2023-03-30 03:57:49,586 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e021c2d717e042aeb52c0e52ead164a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680148667532 2023-03-30 03:57:49,586 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c27264af13dd43aebe786aa6e43cb730, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680148667560 2023-03-30 03:57:49,586 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 304114c6000c47bfb3a64eb529912be2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680148667749 2023-03-30 03:57:49,585 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,585 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/145e81aebdad47a0958e6beb99ef3e96, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/667fbfbf499844fd881066672e98de3f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bbb75f81063f4a6086b1c1cd0b84666b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/490aa8e9d99241e9988760b384e72cec] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=129.1 K 2023-03-30 03:57:49,589 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 145e81aebdad47a0958e6beb99ef3e96, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680148665512 2023-03-30 03:57:49,589 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 667fbfbf499844fd881066672e98de3f, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680148667510 2023-03-30 03:57:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:49,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:49,592 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:49,592 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:49,592 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:49,592 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:49,592 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:49,593 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting bbb75f81063f4a6086b1c1cd0b84666b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680148667560 2023-03-30 03:57:49,594 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 490aa8e9d99241e9988760b384e72cec, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680148667747 2023-03-30 03:57:49,594 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:49,594 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,594 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:49,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:49,596 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,597 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,598 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#B#compaction#330 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:49,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:49,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,602 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:49,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,602 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:49,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,630 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#A#compaction#331 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:49,702 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:49,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:49,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:49,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:49,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:49,705 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:49,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:49,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:49,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,706 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,723 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:49,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:49,724 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,725 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,725 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:49,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:49,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:49,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:49,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:49,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:49,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:49,746 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:49,746 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:49,746 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 03:57:49,747 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:49,747 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:49,747 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:49,747 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:49,747 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:49,747 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:49,763 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d56bbc8791924a1fab6edc397484de0a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d56bbc8791924a1fab6edc397484de0a 2023-03-30 03:57:49,780 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=468, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/58bfd4e277c14181909bb541d0a1f496 2023-03-30 03:57:49,808 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/0ed64d72c775471195f53b8d7080094a 2023-03-30 03:57:49,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:49,899 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:49,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:49,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148729911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34592 deadline: 1680148729912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:49,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148729913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148729914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:49,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:49,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148729919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148730021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148730021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148730021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148730021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,027 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/ff1da3f28e404ee2ab15b715b6b2cc87 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/ff1da3f28e404ee2ab15b715b6b2cc87 2023-03-30 03:57:50,031 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/B of 97a07dc2687acd6de72ca851295ed5b2 into ff1da3f28e404ee2ab15b715b6b2cc87(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:50,031 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:50,031 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/B, priority=12, startTime=1680148669582; duration=0sec 2023-03-30 03:57:50,031 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:50,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,031 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:57:50,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148730026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,032 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:57:50,032 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/C is initiating minor compaction (all files) 2023-03-30 03:57:50,032 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/C in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:50,032 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c02679990ca442ad853b54a2a7b25d0b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d7091aa6f9ee4c03ba75cf5d9910ef99, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3704da99fa5049c0bb1104d5eb2bc32e] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=35.9 K 2023-03-30 03:57:50,033 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c02679990ca442ad853b54a2a7b25d0b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680148667532 2023-03-30 03:57:50,033 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d7091aa6f9ee4c03ba75cf5d9910ef99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680148667560 2023-03-30 03:57:50,033 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3704da99fa5049c0bb1104d5eb2bc32e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680148667749 2023-03-30 03:57:50,038 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#C#compaction#335 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:50,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148730224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148730225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34610 deadline: 1680148730225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148730225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,230 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/c774e26ec715484093a49b4550d58013 2023-03-30 03:57:50,238 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/58bfd4e277c14181909bb541d0a1f496 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/58bfd4e277c14181909bb541d0a1f496 2023-03-30 03:57:50,241 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/58bfd4e277c14181909bb541d0a1f496, entries=150, sequenceid=468, filesize=25.4 K 2023-03-30 03:57:50,242 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/0ed64d72c775471195f53b8d7080094a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/0ed64d72c775471195f53b8d7080094a 2023-03-30 03:57:50,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148730238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,247 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/0ed64d72c775471195f53b8d7080094a, entries=150, sequenceid=468, filesize=11.9 K 2023-03-30 03:57:50,248 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/c774e26ec715484093a49b4550d58013 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c774e26ec715484093a49b4550d58013 2023-03-30 03:57:50,251 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c774e26ec715484093a49b4550d58013, entries=150, sequenceid=468, filesize=11.9 K 2023-03-30 03:57:50,251 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 97a07dc2687acd6de72ca851295ed5b2 in 505ms, sequenceid=468, compaction requested=false 2023-03-30 03:57:50,252 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:50,252 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:50,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:50,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:50,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:50,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:50,253 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:50,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:50,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:50,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:50,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:50,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2945e4e0[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:50,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:50,255 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:50,256 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:50,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,256 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:50,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:50,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:50,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:50,256 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:50,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:50,257 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:50,257 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:50,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:50,260 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:50,260 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:50,260 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:50,260 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:50,260 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:50,262 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,262 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,262 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:50,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:50,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,263 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,263 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:50,266 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:50,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:50,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:50,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:50,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:50,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:50,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:50,267 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:50,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:50,267 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:50,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:50,267 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:50,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:50,267 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,367 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:50,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:50,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:50,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:50,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:50,386 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:50,386 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:50,386 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:50,386 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:50,386 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,386 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:50,387 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:50,387 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:50,387 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,387 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:50,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:50,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:50,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:50,394 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:50,394 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:50,394 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:50,394 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:50,403 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 03:57:50,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:50,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:50,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:50,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:50,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:50,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:50,459 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/5d132486da874c8eafb735c93471ec5f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5d132486da874c8eafb735c93471ec5f 2023-03-30 03:57:50,471 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/C of 97a07dc2687acd6de72ca851295ed5b2 into 5d132486da874c8eafb735c93471ec5f(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:50,471 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:50,471 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/C, priority=13, startTime=1680148669582; duration=0sec 2023-03-30 03:57:50,472 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:50,493 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/6efb494f8b8245cba8f247aaceec4073 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/6efb494f8b8245cba8f247aaceec4073 2023-03-30 03:57:50,497 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/A of 97a07dc2687acd6de72ca851295ed5b2 into 6efb494f8b8245cba8f247aaceec4073(size=25.7 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:50,497 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:50,497 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/A, priority=12, startTime=1680148669582; duration=0sec 2023-03-30 03:57:50,497 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:50,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:50,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148730551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148730552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148730552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148730558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148730558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148730659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148730660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148730661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148730661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148730661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,835 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033089c764b6a02040639e9e82db27442e79 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033089c764b6a02040639e9e82db27442e79 2023-03-30 03:57:50,844 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=482, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/28d58105dd2d4e1685df1ef9f4694fa8 2023-03-30 03:57:50,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148730865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148730865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148730866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148730867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:50,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:50,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148730867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148731172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148731172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148731173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148731173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148731173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,255 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/2e06508f892f4beb978cdf2dbd8b7916 2023-03-30 03:57:51,263 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/a91cc64d76aa4eb29c329b36726c111c 2023-03-30 03:57:51,266 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/28d58105dd2d4e1685df1ef9f4694fa8 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/28d58105dd2d4e1685df1ef9f4694fa8 2023-03-30 03:57:51,269 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/28d58105dd2d4e1685df1ef9f4694fa8, entries=150, sequenceid=482, filesize=25.4 K 2023-03-30 03:57:51,269 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/2e06508f892f4beb978cdf2dbd8b7916 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2e06508f892f4beb978cdf2dbd8b7916 2023-03-30 03:57:51,276 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2e06508f892f4beb978cdf2dbd8b7916, entries=150, sequenceid=482, filesize=11.9 K 2023-03-30 03:57:51,277 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/a91cc64d76aa4eb29c329b36726c111c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a91cc64d76aa4eb29c329b36726c111c 2023-03-30 03:57:51,279 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a91cc64d76aa4eb29c329b36726c111c, entries=150, sequenceid=482, filesize=11.9 K 2023-03-30 03:57:51,280 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 97a07dc2687acd6de72ca851295ed5b2 in 877ms, sequenceid=482, compaction requested=true 2023-03-30 03:57:51,280 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:51,280 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:51,280 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:51,280 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:51,280 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,280 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:51,280 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:51,281 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:51,282 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:51,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:51,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6269a634[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:51,283 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:51,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:51,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:51,283 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:51,283 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:51,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:51,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:51,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:51,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:51,283 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,283 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:51,284 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:51,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:51,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:51,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:51,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:51,286 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:51,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:51,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:51,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:51,286 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:51,288 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,288 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,288 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:51,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:51,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:51,288 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:51,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:51,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:51,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:51,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:51,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:51,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:51,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:51,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:51,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:51,292 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,291 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:51,292 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:51,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:51,392 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:51,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:51,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:51,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:51,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:51,394 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:51,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:51,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:51,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:51,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:51,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,395 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:51,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:51,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:51,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:51,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:51,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:51,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:51,396 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:51,396 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:51,396 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:51,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:51,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:51,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:51,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:51,397 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:57:51,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:51,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:51,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:51,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:51,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:51,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:51,430 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033037960c7610d4458f9de3f3c008031d8e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033037960c7610d4458f9de3f3c008031d8e 2023-03-30 03:57:51,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:51,677 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:51,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148731698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148731699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148731699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148731700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148731710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148731811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148731811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148731811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148731812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148731817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:51,840 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=504, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/b4c3fd62b5bf4ea0b752bff9ee2d8708 2023-03-30 03:57:52,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148732019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148732020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148732020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148732020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148732023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,261 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/1b79b09ba08e48c8a8e88849d78b6bd0 2023-03-30 03:57:52,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148732321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148732327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148732327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148732328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148732329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,671 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/e6ede1636e4a4bc8933b162b5aa9cb89 2023-03-30 03:57:52,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/b4c3fd62b5bf4ea0b752bff9ee2d8708 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/b4c3fd62b5bf4ea0b752bff9ee2d8708 2023-03-30 03:57:52,677 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/b4c3fd62b5bf4ea0b752bff9ee2d8708, entries=150, sequenceid=504, filesize=25.4 K 2023-03-30 03:57:52,677 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/1b79b09ba08e48c8a8e88849d78b6bd0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1b79b09ba08e48c8a8e88849d78b6bd0 2023-03-30 03:57:52,680 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1b79b09ba08e48c8a8e88849d78b6bd0, entries=150, sequenceid=504, filesize=11.9 K 2023-03-30 03:57:52,681 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/e6ede1636e4a4bc8933b162b5aa9cb89 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e6ede1636e4a4bc8933b162b5aa9cb89 2023-03-30 03:57:52,684 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e6ede1636e4a4bc8933b162b5aa9cb89, entries=150, sequenceid=504, filesize=11.9 K 2023-03-30 03:57:52,685 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 97a07dc2687acd6de72ca851295ed5b2 in 1288ms, sequenceid=504, compaction requested=true 2023-03-30 03:57:52,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:52,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:52,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:52,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:52,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:52,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:52,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:52,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:52,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:52,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:52,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@612665b2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:52,687 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:52,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:52,688 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:52,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:52,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:52,689 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:52,689 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:52,689 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:52,690 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:52,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:52,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:52,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:52,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:52,692 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:52,693 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:52,693 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:52,693 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:52,693 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:52,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:52,694 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:52,696 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:52,698 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,699 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,699 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:52,699 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:52,699 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:52,699 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:52,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:52,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:52,699 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:52,700 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:52,700 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:52,700 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:52,700 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:52,700 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:52,700 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:52,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:52,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:52,800 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:52,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:52,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:52,803 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:52,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:52,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:52,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:52,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:52,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,804 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:52,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:52,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:52,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:52,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:52,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:52,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:52,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:52,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:52,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:52,808 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:52,808 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:52,808 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:52,809 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:52,809 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:57:52,809 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:52,809 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:52,809 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:52,809 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:52,809 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:52,809 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:52,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:52,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:52,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148732874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148732877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148732878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148732884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148732885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148732985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148732986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:52,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:52,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148732988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148732993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148732993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148733193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148733194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148733194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148733202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148733203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,227 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330dd6930aa28cb42f89592e1f535ef2fbc to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dd6930aa28cb42f89592e1f535ef2fbc 2023-03-30 03:57:53,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148733501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148733502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148733504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148733512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:53,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148733512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:53,633 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=516, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/ff5ee20c9b70497580655dd22c5c06b8 2023-03-30 03:57:54,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148734007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:54,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148734013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:54,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148734014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:54,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148734019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:54,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148734022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,064 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/143ac5bc83bb4d5d8e7067ec131e2000 2023-03-30 03:57:54,475 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/26c527925cf64f3f81abdaddc8e029e5 2023-03-30 03:57:54,479 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/ff5ee20c9b70497580655dd22c5c06b8 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ff5ee20c9b70497580655dd22c5c06b8 2023-03-30 03:57:54,481 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ff5ee20c9b70497580655dd22c5c06b8, entries=150, sequenceid=516, filesize=25.4 K 2023-03-30 03:57:54,481 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/143ac5bc83bb4d5d8e7067ec131e2000 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/143ac5bc83bb4d5d8e7067ec131e2000 2023-03-30 03:57:54,484 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/143ac5bc83bb4d5d8e7067ec131e2000, entries=150, sequenceid=516, filesize=11.9 K 2023-03-30 03:57:54,485 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/26c527925cf64f3f81abdaddc8e029e5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/26c527925cf64f3f81abdaddc8e029e5 2023-03-30 03:57:54,487 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/26c527925cf64f3f81abdaddc8e029e5, entries=150, sequenceid=516, filesize=11.9 K 2023-03-30 03:57:54,488 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 97a07dc2687acd6de72ca851295ed5b2 in 1679ms, sequenceid=516, compaction requested=true 2023-03-30 03:57:54,488 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:54,488 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:54,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:54,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:54,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:54,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:54,489 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:54,489 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:54,489 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:54,489 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:54,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:54,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@271051d4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:54,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:54,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:54,490 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:54,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:54,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:54,490 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:54,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:54,491 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,491 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:54,491 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:54,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:54,491 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:54,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:54,493 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:54,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:54,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:54,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:54,493 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:54,494 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,494 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,494 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:54,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:54,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:54,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:54,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:54,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:54,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:54,498 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:54,498 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:54,498 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:54,598 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:54,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:54,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:54,601 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:54,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:54,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:54,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:54,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,601 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:54,602 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:54,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:54,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:54,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:54,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:54,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:54,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:54,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:54,603 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:54,603 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:54,603 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:57:54,603 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:55,013 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033012ee0959cbae4e2e979639262c1bf72d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033012ee0959cbae4e2e979639262c1bf72d 2023-03-30 03:57:55,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:55,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:55,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148735035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148735036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148735037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148735038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148735040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148735144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148735144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148735144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148735144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148735148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148735352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148735353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148735353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148735353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148735354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,420 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=536, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/ae649dba77124c788791adc245565963 2023-03-30 03:57:55,428 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/c6e11107c7e7489e864c94d2786332b9 2023-03-30 03:57:55,444 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/dc77442af7b24d7085950aca67879269 2023-03-30 03:57:55,455 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/ae649dba77124c788791adc245565963 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ae649dba77124c788791adc245565963 2023-03-30 03:57:55,458 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ae649dba77124c788791adc245565963, entries=150, sequenceid=536, filesize=25.4 K 2023-03-30 03:57:55,458 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/c6e11107c7e7489e864c94d2786332b9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c6e11107c7e7489e864c94d2786332b9 2023-03-30 03:57:55,462 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c6e11107c7e7489e864c94d2786332b9, entries=150, sequenceid=536, filesize=11.9 K 2023-03-30 03:57:55,463 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/dc77442af7b24d7085950aca67879269 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dc77442af7b24d7085950aca67879269 2023-03-30 03:57:55,467 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dc77442af7b24d7085950aca67879269, entries=150, sequenceid=536, filesize=11.9 K 2023-03-30 03:57:55,467 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 97a07dc2687acd6de72ca851295ed5b2 in 864ms, sequenceid=536, compaction requested=true 2023-03-30 03:57:55,468 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:55,468 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:55,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:55,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:55,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:55,470 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:55,471 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:55,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:55,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:55,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:55,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:55,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2174c332[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:55,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:55,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:55,473 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:55,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:55,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:55,473 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,473 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:55,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:55,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:55,474 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:55,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:55,475 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:55,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:55,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:55,476 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:55,476 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:55,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:55,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:55,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:55,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:55,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:55,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:55,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:55,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:55,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:55,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:55,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:55,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:55,480 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:55,480 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:55,480 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,481 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:55,481 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:55,481 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,580 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:55,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:55,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:55,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:55,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:55,594 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:55,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:55,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:55,594 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:55,595 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,595 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:55,595 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,595 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:55,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:55,596 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,596 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:55,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:55,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:55,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:55,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:55,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:55,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:55,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:55,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:55,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:55,597 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:55,597 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:55,597 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:55,598 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:55,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:55,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:55,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148735687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148735688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148735689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148735690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148735696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148735802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148735802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148735802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148735803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:55,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148735811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148736011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148736011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148736011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148736013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,023 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330fd9e1e9204d84287baf3828f45193f99 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330fd9e1e9204d84287baf3828f45193f99 2023-03-30 03:57:56,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148736017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,028 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=548, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/eded85f1dc3b48e7859e5b7cb6af7b56 2023-03-30 03:57:56,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148736316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148736316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148736317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148736319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148736327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,437 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/696fad92a4e44e93ababda46e3e47151 2023-03-30 03:57:56,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148736824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148736825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148736826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148736826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:56,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148736836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,847 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/8b114e2315054219867905d559fbb68e 2023-03-30 03:57:56,850 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/eded85f1dc3b48e7859e5b7cb6af7b56 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/eded85f1dc3b48e7859e5b7cb6af7b56 2023-03-30 03:57:56,852 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/eded85f1dc3b48e7859e5b7cb6af7b56, entries=150, sequenceid=548, filesize=25.4 K 2023-03-30 03:57:56,852 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/696fad92a4e44e93ababda46e3e47151 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/696fad92a4e44e93ababda46e3e47151 2023-03-30 03:57:56,855 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/696fad92a4e44e93ababda46e3e47151, entries=150, sequenceid=548, filesize=11.9 K 2023-03-30 03:57:56,856 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/8b114e2315054219867905d559fbb68e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8b114e2315054219867905d559fbb68e 2023-03-30 03:57:56,859 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8b114e2315054219867905d559fbb68e, entries=150, sequenceid=548, filesize=11.9 K 2023-03-30 03:57:56,859 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 97a07dc2687acd6de72ca851295ed5b2 in 1262ms, sequenceid=548, compaction requested=true 2023-03-30 03:57:56,860 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:56,860 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:56,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:56,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:56,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:56,860 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:56,861 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:56,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:56,861 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:56,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:56,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e83892e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:56,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:56,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:56,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:56,864 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:56,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:56,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:56,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:56,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:56,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:56,865 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:56,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:56,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:56,867 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:56,867 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:56,867 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:56,867 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:56,867 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:56,868 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,868 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,868 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:56,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,868 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:56,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:56,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:56,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:56,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:56,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:56,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:56,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:56,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:56,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:56,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:56,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,872 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:56,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:56,873 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:56,873 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:56,973 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:56,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:56,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:56,976 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:56,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:56,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:56,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:56,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:56,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:56,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:56,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:56,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:56,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:56,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:56,982 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:56,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:56,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:56,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:56,983 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:56,983 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:56,983 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:57:56,983 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:56,984 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:56,984 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:56,984 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:56,984 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:56,984 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:57,407 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d0b2bf30e8ba486788b8701f201d9eb6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d0b2bf30e8ba486788b8701f201d9eb6 2023-03-30 03:57:57,419 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=568, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/8e378c2834794c7ab4a9147aca7a0057 2023-03-30 03:57:57,433 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/1c8f9ab6a79a444386922ed4a527d587 2023-03-30 03:57:57,444 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/79692eb843184f05874f7e57718e67fe 2023-03-30 03:57:57,448 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/8e378c2834794c7ab4a9147aca7a0057 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/8e378c2834794c7ab4a9147aca7a0057 2023-03-30 03:57:57,451 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/8e378c2834794c7ab4a9147aca7a0057, entries=150, sequenceid=568, filesize=25.4 K 2023-03-30 03:57:57,451 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/1c8f9ab6a79a444386922ed4a527d587 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1c8f9ab6a79a444386922ed4a527d587 2023-03-30 03:57:57,454 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1c8f9ab6a79a444386922ed4a527d587, entries=150, sequenceid=568, filesize=11.9 K 2023-03-30 03:57:57,455 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/79692eb843184f05874f7e57718e67fe as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/79692eb843184f05874f7e57718e67fe 2023-03-30 03:57:57,469 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/79692eb843184f05874f7e57718e67fe, entries=150, sequenceid=568, filesize=11.9 K 2023-03-30 03:57:57,471 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 97a07dc2687acd6de72ca851295ed5b2 in 488ms, sequenceid=568, compaction requested=true 2023-03-30 03:57:57,471 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:57,471 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:57,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:57,472 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:57,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c5a76b9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:57,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:57,474 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,475 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:57,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:57,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:57,475 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,475 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:57,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:57,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:57,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:57,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:57,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:57,490 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:57,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:57,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:57,490 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:57,490 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:57,491 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,491 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,492 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:57,492 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,493 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,495 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:57,495 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:57,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:57,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:57,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:57,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:57,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:57,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:57,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,496 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:57,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:57,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,496 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:57,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,496 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,596 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:57,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:57,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:57,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:57,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:57,599 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:57,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:57,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:57,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:57,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,600 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:57,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:57,600 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,600 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:57,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:57,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:57,601 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:57,601 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:57,601 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:57,602 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:57,602 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:57,602 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:57,602 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:57,604 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,605 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,605 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:57,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e10217[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:57,606 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:57,606 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,607 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:57,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:57,607 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:57,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,607 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:57,608 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:57,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,608 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:57,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:57,610 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:57,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:57,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:57,610 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:57,610 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:57,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,611 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:57,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,615 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:57,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:57,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:57,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:57,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:57,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:57,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,616 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:57,616 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:57,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:57,716 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:57,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:57,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:57,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:57,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:57,732 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:57,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:57,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:57,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:57,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,733 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:57,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:57,734 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,734 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:57,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:57,738 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:57,738 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:57,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:57,739 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:57,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:57,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:57,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:57,739 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:57,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,740 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:57,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8b234a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:57,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:57,742 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,742 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,743 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,743 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:57,743 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:57,743 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:57,743 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:57,743 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:57,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:57,746 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:57,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:57,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:57,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:57,746 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:57,747 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,747 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:57,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,747 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:57,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:57,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:57,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:57,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:57,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:57,752 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:57,752 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:57,852 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:57,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:57,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:57,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:57,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:57,855 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:57,855 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:57,855 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:57,855 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:57,856 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,856 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,857 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:57,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:57,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:57,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:57,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:57,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:57,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:57,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:57,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:57,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:57,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:57,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:57,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:57,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:57,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:57,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:57,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:57,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:57,873 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,873 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,873 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:57,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c530f4c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:57,875 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:57,876 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,880 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:57,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:57,880 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:57,880 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,881 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:57,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:57,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:57,881 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:57,881 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:57,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:57,884 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:57,884 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:57,884 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:57,884 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:57,884 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:57,884 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,884 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:57,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,885 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:57,885 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,886 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,893 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:57,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:57,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:57,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:57,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:57,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:57,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:57,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:57,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:57,894 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:57,895 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:57,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:57:57,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:57:57,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:57,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:57,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:57,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:57,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:57,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:57:57,995 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:57,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:57,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:57,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:57,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:57,997 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:57,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:57,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:57,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:57,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:57,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:57,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:57,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:57,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:57,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:57,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:58,000 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,000 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:58,000 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:58,000 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:58,000 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:58,001 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,001 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,001 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,001 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:58,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b343c1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:58,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:58,002 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,003 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,004 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:58,004 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:58,004 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:58,004 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:58,004 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,005 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,005 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,005 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680148677985, max timestamp=1680148678005Num uniques 859; 2023-03-30 03:57:58,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 03:57:58,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680148677985, max timestamp=1680148678005Num uniques 859; 2023-03-30 03:57:58,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 03:57:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:58,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:58,013 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:58,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:58,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:58,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:58,013 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:58,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,015 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680148677985, max timestamp=1680148678005Num uniques 859; 2023-03-30 03:57:58,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 03:57:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,015 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,016 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,018 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,018 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,018 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,018 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,018 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,019 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,019 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,019 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,019 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,019 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:58,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:58,019 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:58,019 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:58,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:58,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148738035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148738035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148738036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148738046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148738048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,119 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:58,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:58,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:58,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:58,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:58,123 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:58,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:58,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:58,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,123 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,124 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,124 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:58,126 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:58,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:58,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:58,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:58,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:58,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:58,127 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:58,127 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,127 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:58,128 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:58,128 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:58,128 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:58,129 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:58,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b3141a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:58,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:58,130 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,131 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:58,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:58,131 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,131 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:58,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:58,132 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:58,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:58,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:58,136 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:58,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:58,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:58,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:58,136 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:58,139 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,139 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,139 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,140 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,141 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,142 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,142 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,143 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:58,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:58,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:58,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,143 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:58,143 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:58,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,143 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148738149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148738149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148738150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148738156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148738168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,243 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:58,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:58,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:58,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:58,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:58,246 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:58,246 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,246 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:58,246 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:58,247 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,247 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,247 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,247 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,248 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,248 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:58,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:58,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:58,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:58,249 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:58,249 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,249 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:58,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:58,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:58,250 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,250 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:58,250 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:58,250 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:58,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:58,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:58,250 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:58,251 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,251 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,252 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:58,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3585f475[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:58,253 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:58,253 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,253 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,253 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:58,253 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:58,253 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:58,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,254 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:58,254 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:58,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:58,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:58,258 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:58,258 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:58,258 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:58,258 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:58,258 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:58,259 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,259 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,259 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,260 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,272 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:58,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:58,272 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,272 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,272 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:58,272 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:58,272 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,272 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,272 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,358 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330c9f66efd283942c386dd46f6dcfbd2ad to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c9f66efd283942c386dd46f6dcfbd2ad 2023-03-30 03:57:58,365 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=582, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/e8970915c96b439bb79f6c1a3e6e04c5 2023-03-30 03:57:58,372 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:58,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:58,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:58,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:58,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:58,376 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:58,376 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,376 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:58,376 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:58,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148738366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148738366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148738367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148738368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,379 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:58,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:58,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:58,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:58,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:58,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:58,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:58,382 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,382 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:58,382 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:58,382 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:58,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:58,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:58,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:58,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,383 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:58,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41d8a4e6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:58,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:58,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,385 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:58,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:58,385 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:58,385 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,385 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:58,387 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,387 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:58,387 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,387 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:58,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:58,389 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:58,389 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:58,389 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:58,389 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:58,389 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:58,390 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,390 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,390 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,391 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148738387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,392 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,397 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,397 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,397 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,397 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,397 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,397 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:58,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,397 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,398 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,398 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:58,397 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,398 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:58,398 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:58,498 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:58,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:58,510 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:58,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:58,510 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:58,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,511 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,511 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,512 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,512 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:58,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:58,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:58,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:58,519 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,519 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:58,519 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:58,519 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:58,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:58,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,523 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,523 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,523 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:58,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@380c5a57[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:58,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:58,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:58,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:58,525 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:58,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:58,526 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:58,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:58,527 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:58,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:58,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:58,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:58,528 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:58,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,529 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,530 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,531 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:58,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:58,532 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:58,532 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:58,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,632 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:58,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:58,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:58,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:58,636 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:58,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:58,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:58,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:58,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:58,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:58,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:58,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:58,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:58,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:58,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:58,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:58,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:58,647 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,647 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,647 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,647 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:58,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32959a88[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:58,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:58,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,656 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,656 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,656 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:58,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:58,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:58,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:58,663 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:58,663 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:58,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:58,666 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:58,666 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:58,666 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:58,666 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:58,666 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:58,673 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,673 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,673 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,673 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,676 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:58,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:58,680 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,680 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:58,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,681 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:58,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148738678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148738678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148738678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148738679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:58,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148738695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,775 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/5844ffb4eeea481f97743865cdd45a33 2023-03-30 03:57:58,781 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:58,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:58,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:58,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:58,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:58,784 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:58,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:58,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:58,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,785 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,785 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,785 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:58,787 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:58,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:58,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:58,788 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,788 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:58,788 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:58,788 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:58,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:58,789 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,790 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,790 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:58,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2829e6f3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:58,791 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:58,791 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,791 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:58,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:58,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:58,792 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,792 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:58,793 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,793 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:58,793 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,793 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:58,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:58,795 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:58,795 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:58,795 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:58,795 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:58,795 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:58,796 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,796 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,796 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,798 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,800 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:58,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:58,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:58,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,801 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,801 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:58,801 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:58,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,901 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:58,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:58,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:58,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:58,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:58,905 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:58,905 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,905 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:58,905 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:58,906 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,906 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,906 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,906 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,907 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,907 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:58,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:58,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:58,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:58,910 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:58,910 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:58,910 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:58,913 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:58,913 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:58,913 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,913 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:58,914 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:58,914 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:58,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:58,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:58,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:58,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:58,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,915 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,915 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:58,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd29884[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:58,916 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:58,916 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,916 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:58,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:58,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:58,917 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,917 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:58,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:58,917 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:58,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:58,919 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:58,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:58,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:58,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:58,919 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:58,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:58,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:58,921 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,923 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:58,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:58,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:58,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:58,925 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:58,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:58,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:58,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:58,925 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:58,925 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:58,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:58,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:58,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,025 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:59,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:59,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:59,034 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:59,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:59,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:59,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:59,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,035 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,035 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:59,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:59,035 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,035 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:59,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:59,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:59,036 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:59,036 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:59,037 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:59,037 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:59,037 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:59,037 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:59,038 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,038 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:59,038 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,038 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:59,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:59,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76d6b1ab[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:59,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:59,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,041 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:59,042 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,042 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:59,042 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:59,043 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:59,044 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:59,044 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:59,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:59,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:59,045 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:59,046 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,046 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,046 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:59,046 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,051 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:59,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:59,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:59,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:59,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:59,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:59,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:59,052 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:59,054 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:59,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:59,154 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:59,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:59,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:59,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:59,157 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:59,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:59,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:59,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:59,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,157 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,164 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:59,164 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:59,165 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,165 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:59,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:59,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:59,166 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:59,166 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:59,166 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:59,166 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:59,166 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:59,166 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:59,166 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:59,166 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:57:59,166 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:59,166 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:57:59,166 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:57:59,166 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,167 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:57:59,167 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:57:59,172 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:59,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:59,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:57:59,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68092423[Count = 0] remaining members to acquire global barrier 2023-03-30 03:57:59,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:57:59,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:57:59,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:57:59,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:57:59,181 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:57:59,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:57:59,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/5a591466b3ee4cc3a5049461ae717715 2023-03-30 03:57:59,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/e8970915c96b439bb79f6c1a3e6e04c5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e8970915c96b439bb79f6c1a3e6e04c5 2023-03-30 03:57:59,189 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:59,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:57:59,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:59,189 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:57:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:57:59,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:57:59,198 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:57:59,198 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:57:59,198 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:57:59,198 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:57:59,198 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:57:59,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:59,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148739191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:59,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148739192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,201 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:57:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:57:59,201 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,201 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:59,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:57:59,202 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:57:59,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:57:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148739192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e8970915c96b439bb79f6c1a3e6e04c5, entries=250, sequenceid=582, filesize=39.2 K 2023-03-30 03:57:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148739193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:59,209 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:57:59,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:59,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:57:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:57:59,209 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,209 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,209 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,209 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:57:59,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:57:59,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:57:59,209 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,210 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:57:59,210 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,209 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:57:59,210 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:57:59,210 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:57:59,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:57:59,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/5844ffb4eeea481f97743865cdd45a33 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/5844ffb4eeea481f97743865cdd45a33 2023-03-30 03:57:59,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/5844ffb4eeea481f97743865cdd45a33, entries=100, sequenceid=582, filesize=9.5 K 2023-03-30 03:57:59,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/5a591466b3ee4cc3a5049461ae717715 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5a591466b3ee4cc3a5049461ae717715 2023-03-30 03:57:59,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:57:59,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148739206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5a591466b3ee4cc3a5049461ae717715, entries=100, sequenceid=582, filesize=9.5 K 2023-03-30 03:57:59,223 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 97a07dc2687acd6de72ca851295ed5b2 in 1276ms, sequenceid=582, compaction requested=true 2023-03-30 03:57:59,223 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:59,223 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:59,223 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-30 03:57:59,223 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-30 03:57:59,223 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:59,223 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:59,227 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 248768 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-30 03:57:59,227 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/A is initiating minor compaction (all files) 2023-03-30 03:57:59,227 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/A in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:59,227 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/6efb494f8b8245cba8f247aaceec4073, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/58bfd4e277c14181909bb541d0a1f496, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/28d58105dd2d4e1685df1ef9f4694fa8, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/b4c3fd62b5bf4ea0b752bff9ee2d8708, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ff5ee20c9b70497580655dd22c5c06b8, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ae649dba77124c788791adc245565963, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/eded85f1dc3b48e7859e5b7cb6af7b56, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/8e378c2834794c7ab4a9147aca7a0057, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e8970915c96b439bb79f6c1a3e6e04c5] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=242.9 K 2023-03-30 03:57:59,227 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107504 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-30 03:57:59,228 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/B is initiating minor compaction (all files) 2023-03-30 03:57:59,228 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/B in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:59,228 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/ff1da3f28e404ee2ab15b715b6b2cc87, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/0ed64d72c775471195f53b8d7080094a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2e06508f892f4beb978cdf2dbd8b7916, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1b79b09ba08e48c8a8e88849d78b6bd0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/143ac5bc83bb4d5d8e7067ec131e2000, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c6e11107c7e7489e864c94d2786332b9, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/696fad92a4e44e93ababda46e3e47151, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1c8f9ab6a79a444386922ed4a527d587, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/5844ffb4eeea481f97743865cdd45a33] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=105.0 K 2023-03-30 03:57:59,228 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ff1da3f28e404ee2ab15b715b6b2cc87, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680148667749 2023-03-30 03:57:59,228 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 0ed64d72c775471195f53b8d7080094a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680148668727 2023-03-30 03:57:59,229 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2e06508f892f4beb978cdf2dbd8b7916, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1680148669912 2023-03-30 03:57:59,229 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1b79b09ba08e48c8a8e88849d78b6bd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680148670551 2023-03-30 03:57:59,229 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 143ac5bc83bb4d5d8e7067ec131e2000, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1680148671693 2023-03-30 03:57:59,230 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c6e11107c7e7489e864c94d2786332b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680148672877 2023-03-30 03:57:59,231 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 696fad92a4e44e93ababda46e3e47151, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680148675036 2023-03-30 03:57:59,231 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 6efb494f8b8245cba8f247aaceec4073, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680148667749 2023-03-30 03:57:59,232 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1c8f9ab6a79a444386922ed4a527d587, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680148675688 2023-03-30 03:57:59,232 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 58bfd4e277c14181909bb541d0a1f496, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680148668727 2023-03-30 03:57:59,232 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5844ffb4eeea481f97743865cdd45a33, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680148677929 2023-03-30 03:57:59,232 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 28d58105dd2d4e1685df1ef9f4694fa8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1680148669912 2023-03-30 03:57:59,233 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting b4c3fd62b5bf4ea0b752bff9ee2d8708, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680148670551 2023-03-30 03:57:59,233 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting ff5ee20c9b70497580655dd22c5c06b8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1680148671693 2023-03-30 03:57:59,234 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting ae649dba77124c788791adc245565963, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680148672877 2023-03-30 03:57:59,234 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting eded85f1dc3b48e7859e5b7cb6af7b56, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680148675036 2023-03-30 03:57:59,235 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 8e378c2834794c7ab4a9147aca7a0057, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680148675688 2023-03-30 03:57:59,235 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting e8970915c96b439bb79f6c1a3e6e04c5, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680148677923 2023-03-30 03:57:59,260 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#B#compaction#357 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:59,284 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#A#compaction#358 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:59,293 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/1a46aedbe51743a0bfef192fce8c290f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1a46aedbe51743a0bfef192fce8c290f 2023-03-30 03:57:59,303 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/B of 97a07dc2687acd6de72ca851295ed5b2 into 1a46aedbe51743a0bfef192fce8c290f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:59,304 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:59,304 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/B, priority=7, startTime=1680148679223; duration=0sec 2023-03-30 03:57:59,304 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:57:59,304 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-30 03:57:59,307 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107504 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-30 03:57:59,307 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/C is initiating minor compaction (all files) 2023-03-30 03:57:59,307 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/C in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:59,307 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5d132486da874c8eafb735c93471ec5f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c774e26ec715484093a49b4550d58013, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a91cc64d76aa4eb29c329b36726c111c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e6ede1636e4a4bc8933b162b5aa9cb89, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/26c527925cf64f3f81abdaddc8e029e5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dc77442af7b24d7085950aca67879269, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8b114e2315054219867905d559fbb68e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/79692eb843184f05874f7e57718e67fe, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5a591466b3ee4cc3a5049461ae717715] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=105.0 K 2023-03-30 03:57:59,308 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5d132486da874c8eafb735c93471ec5f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680148667749 2023-03-30 03:57:59,309 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c774e26ec715484093a49b4550d58013, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680148668727 2023-03-30 03:57:59,309 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a91cc64d76aa4eb29c329b36726c111c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1680148669912 2023-03-30 03:57:59,309 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e6ede1636e4a4bc8933b162b5aa9cb89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680148670551 2023-03-30 03:57:59,310 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 26c527925cf64f3f81abdaddc8e029e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1680148671693 2023-03-30 03:57:59,310 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:57:59,310 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting dc77442af7b24d7085950aca67879269, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680148672877 2023-03-30 03:57:59,311 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8b114e2315054219867905d559fbb68e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680148675036 2023-03-30 03:57:59,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:57:59,311 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 79692eb843184f05874f7e57718e67fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680148675688 2023-03-30 03:57:59,312 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5a591466b3ee4cc3a5049461ae717715, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680148677929 2023-03-30 03:57:59,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:57:59,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:57:59,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:57:59,318 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:57:59,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:59,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:57:59,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:57:59,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,319 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:57:59,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:57:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:57:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:57:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:57:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:57:59,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:57:59,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:57:59,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:57:59,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:57:59,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:57:59,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:57:59,323 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:57:59,323 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:57:59,326 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:57:59,326 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:57:59,326 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-03-30 03:57:59,326 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:57:59,326 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:59,327 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:57:59,327 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:59,327 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:57:59,327 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:57:59,332 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#C#compaction#359 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:57:59,358 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d58dde4180e6472eb9bcd7a2cc7ed90c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d58dde4180e6472eb9bcd7a2cc7ed90c 2023-03-30 03:57:59,720 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/bfe80dcdc2e740d290df07f24154dd5d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bfe80dcdc2e740d290df07f24154dd5d 2023-03-30 03:57:59,724 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/A of 97a07dc2687acd6de72ca851295ed5b2 into bfe80dcdc2e740d290df07f24154dd5d(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:59,724 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:59,724 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/A, priority=7, startTime=1680148679223; duration=0sec 2023-03-30 03:57:59,724 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:59,750 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/bc4e8fedb5ba454292961c5e87417335 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bc4e8fedb5ba454292961c5e87417335 2023-03-30 03:57:59,755 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/C of 97a07dc2687acd6de72ca851295ed5b2 into bc4e8fedb5ba454292961c5e87417335(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:57:59,755 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:57:59,755 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/C, priority=7, startTime=1680148679223; duration=0sec 2023-03-30 03:57:59,755 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:57:59,762 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=616, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/5266cda90b67438bb306ad8620219e60 2023-03-30 03:57:59,770 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/3376bd11ba8b441db7f6e427bc0bf1d0 2023-03-30 03:58:00,179 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/e1f62204bce5496f8c45b34c7670a052 2023-03-30 03:58:00,185 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/5266cda90b67438bb306ad8620219e60 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/5266cda90b67438bb306ad8620219e60 2023-03-30 03:58:00,188 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/5266cda90b67438bb306ad8620219e60, entries=150, sequenceid=616, filesize=25.4 K 2023-03-30 03:58:00,189 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/3376bd11ba8b441db7f6e427bc0bf1d0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/3376bd11ba8b441db7f6e427bc0bf1d0 2023-03-30 03:58:00,193 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/3376bd11ba8b441db7f6e427bc0bf1d0, entries=150, sequenceid=616, filesize=11.9 K 2023-03-30 03:58:00,194 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/e1f62204bce5496f8c45b34c7670a052 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e1f62204bce5496f8c45b34c7670a052 2023-03-30 03:58:00,214 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e1f62204bce5496f8c45b34c7670a052, entries=150, sequenceid=616, filesize=11.9 K 2023-03-30 03:58:00,215 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=0 B/0 for 97a07dc2687acd6de72ca851295ed5b2 in 889ms, sequenceid=616, compaction requested=false 2023-03-30 03:58:00,215 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:00,216 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:00,217 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,217 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,217 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,217 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff059af[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:00,219 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:00,219 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,220 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,220 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:00,220 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:00,220 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:00,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,220 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:00,221 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:00,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:00,223 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:00,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:00,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:00,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:00,223 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:00,225 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,225 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,225 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,225 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,227 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,228 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,229 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,230 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,230 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,230 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,230 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:00,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:00,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:00,230 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,230 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,230 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,230 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,230 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:00,230 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:00,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:58:00,287 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:00,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:58:00,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:00,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:58:00,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:00,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:58:00,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:00,304 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330513e431f20dd4a9dbfc1cd664710ae61 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330513e431f20dd4a9dbfc1cd664710ae61 2023-03-30 03:58:00,331 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:00,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:00,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:00,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:00,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:00,335 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:00,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:00,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:00,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,336 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,336 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,337 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,337 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:00,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148680325, max timestamp=1680148680335Num uniques 859; 2023-03-30 03:58:00,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:00,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:00,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:00,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:00,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:00,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148680325, max timestamp=1680148680335Num uniques 859; 2023-03-30 03:58:00,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:00,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148680325, max timestamp=1680148680335Num uniques 859; 2023-03-30 03:58:00,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:00,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:00,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:00,340 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,340 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:00,340 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:00,340 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:00,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:00,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:00,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:00,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,347 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,347 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,347 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:00,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37def66d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:00,349 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:00,349 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,356 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,356 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:00,356 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:00,356 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:00,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:00,363 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,364 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,364 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,364 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:00,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:00,366 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:00,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:00,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:00,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:00,366 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:00,373 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148740365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,378 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,378 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,381 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,383 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:00,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:00,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,383 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,384 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:00,384 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:00,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148740374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148740375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148740376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148740377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148740478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,484 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:00,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:00,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:00,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:00,494 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:00,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:00,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:00,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,494 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,495 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:00,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:00,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148740486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:00,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148740488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148740488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148740488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:00,500 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:00,500 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,500 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:00,506 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:00,506 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:00,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:00,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:00,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:00,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:00,519 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:00,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744642d4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:00,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:00,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,523 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,523 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:00,523 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:00,523 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:00,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:00,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:00,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,524 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,524 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:00,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:00,526 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:00,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:00,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:00,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:00,526 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:00,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,530 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:00,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:00,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,531 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:00,531 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:00,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:00,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,631 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:00,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:00,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:00,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:00,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:00,634 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:00,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:00,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:00,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,636 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:00,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:00,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:00,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:00,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:00,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:00,646 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,646 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:00,646 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:00,646 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:00,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:00,648 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,648 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,648 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,648 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:00,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55cf4843[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:00,651 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:00,651 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,652 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,652 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:00,652 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:00,652 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,652 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:00,652 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:00,652 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:00,652 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,653 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,653 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:00,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:00,654 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:00,654 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:00,654 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:00,654 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:00,654 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:00,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,656 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,657 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,658 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,658 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,658 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:00,658 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,658 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:00,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:00,659 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,659 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,659 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,659 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,659 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,659 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:00,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,659 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148740684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148740697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148740697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148740697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148740698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,712 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=628, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/c2d75fd00950431190f2517a5a0fef45 2023-03-30 03:58:00,759 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:00,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:00,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:00,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:00,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:00,766 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:00,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:00,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:00,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,766 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,767 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,767 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,767 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,768 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:00,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:00,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:00,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:00,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,770 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:00,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:00,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:00,774 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,774 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:00,774 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:00,774 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:00,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:00,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:00,774 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:00,781 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,781 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,782 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,782 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:00,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@587f5e74[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:00,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:00,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,783 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,784 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:00,784 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:00,784 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:00,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:00,784 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:00,784 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,784 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:00,785 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:00,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:00,789 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:00,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:00,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:00,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:00,790 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:00,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,797 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,800 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,801 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,804 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,804 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:00,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:00,806 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:00,806 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:00,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:00,906 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:00,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:00,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:00,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:00,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:00,908 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:00,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:00,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:00,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:00,912 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,912 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:00,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:00,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:00,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:00,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:00,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:00,914 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,914 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:00,914 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:00,914 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:00,914 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:00,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,916 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:00,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d2b688[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:00,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:00,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,920 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:00,920 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:00,920 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:00,920 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,920 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:00,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,920 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,920 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,921 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:00,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:00,923 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:00,923 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:00,923 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:00,923 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:00,923 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:00,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:00,925 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,926 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:00,928 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:00,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:00,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:00,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:00,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:00,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:00,929 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:00,929 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:00,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:00,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:00,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148740989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148741009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148741009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148741009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148741010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,029 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:01,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:01,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:01,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:01,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:01,038 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:01,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:01,038 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:01,039 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,039 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,040 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,042 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,042 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:01,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:01,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:01,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:01,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:01,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:01,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:01,062 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:01,064 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,064 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:01,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e4407a4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:01,067 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:01,067 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,068 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,068 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,068 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:01,068 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:01,069 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:01,069 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:01,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:01,072 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:01,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:01,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:01,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:01,073 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:01,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,074 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,075 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,077 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:01,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:01,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:01,080 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:01,080 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:01,080 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,081 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:01,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:01,133 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/acb26dde7b014c018bb90cb7b2866dac 2023-03-30 03:58:01,180 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:01,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:01,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:01,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:01,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:01,186 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:01,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:01,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:01,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:01,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:01,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:01,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:01,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,190 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:01,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:01,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:01,194 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,194 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:01,194 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:01,194 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:01,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:01,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:01,194 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:01,198 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:01,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71e6c17f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:01,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:01,199 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,200 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,200 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:01,200 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:01,200 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:01,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,200 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:01,202 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,202 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:01,202 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:01,202 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:01,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:01,204 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:01,204 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:01,204 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:01,204 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:01,204 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:01,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,207 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:01,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:01,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,208 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:01,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:01,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,208 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:01,208 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:01,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:01,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:01,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,209 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,308 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:01,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:01,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:01,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:01,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:01,318 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:01,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:01,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:01,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,318 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,319 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:01,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:01,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:01,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:01,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:01,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:01,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:01,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:01,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:01,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:01,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:01,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:01,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:01,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,339 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:01,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5348adc9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:01,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:01,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,347 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:01,347 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:01,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:01,347 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:01,348 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,348 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:01,348 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:01,348 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:01,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:01,350 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:01,350 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:01,350 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:01,350 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:01,350 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:01,351 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,351 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,351 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:01,352 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,352 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:01,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:01,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,354 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:01,354 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:01,354 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,354 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:01,454 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:01,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:01,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:01,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:01,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:01,457 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:01,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:01,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:01,458 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,458 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,459 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,461 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,461 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:01,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:01,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:01,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:01,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:01,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:01,466 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:01,466 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:01,467 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,467 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,468 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:01,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea327de[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:01,471 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:01,471 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,471 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:01,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:01,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:01,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,472 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:01,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:01,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:01,473 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:01,474 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:01,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:01,480 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:01,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:01,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:01,480 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:01,480 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:01,481 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,481 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:01,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:01,481 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,482 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,485 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:01,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:01,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:01,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,485 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:01,486 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:01,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,485 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:01,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:01,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:01,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:01,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148741497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148741515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148741516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148741518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:01,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148741520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/8c433e26307640e29620d93d354240f2 2023-03-30 03:58:01,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/c2d75fd00950431190f2517a5a0fef45 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/c2d75fd00950431190f2517a5a0fef45 2023-03-30 03:58:01,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/c2d75fd00950431190f2517a5a0fef45, entries=350, sequenceid=628, filesize=52.9 K 2023-03-30 03:58:01,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/acb26dde7b014c018bb90cb7b2866dac as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/acb26dde7b014c018bb90cb7b2866dac 2023-03-30 03:58:01,559 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/acb26dde7b014c018bb90cb7b2866dac, entries=150, sequenceid=628, filesize=11.9 K 2023-03-30 03:58:01,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/8c433e26307640e29620d93d354240f2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8c433e26307640e29620d93d354240f2 2023-03-30 03:58:01,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8c433e26307640e29620d93d354240f2, entries=150, sequenceid=628, filesize=11.9 K 2023-03-30 03:58:01,563 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 97a07dc2687acd6de72ca851295ed5b2 in 1276ms, sequenceid=628, compaction requested=true 2023-03-30 03:58:01,563 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:01,563 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:01,563 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:01,563 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:01,563 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:01,563 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:01,564 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:01,564 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:01,564 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/A is initiating minor compaction (all files) 2023-03-30 03:58:01,564 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/B is initiating minor compaction (all files) 2023-03-30 03:58:01,564 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/A in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,564 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/B in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,564 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bfe80dcdc2e740d290df07f24154dd5d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/5266cda90b67438bb306ad8620219e60, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/c2d75fd00950431190f2517a5a0fef45] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=104.4 K 2023-03-30 03:58:01,564 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1a46aedbe51743a0bfef192fce8c290f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/3376bd11ba8b441db7f6e427bc0bf1d0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/acb26dde7b014c018bb90cb7b2866dac] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=36.3 K 2023-03-30 03:58:01,564 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting bfe80dcdc2e740d290df07f24154dd5d, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680148675689 2023-03-30 03:58:01,565 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1a46aedbe51743a0bfef192fce8c290f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680148675689 2023-03-30 03:58:01,565 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5266cda90b67438bb306ad8620219e60, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680148678035 2023-03-30 03:58:01,565 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3376bd11ba8b441db7f6e427bc0bf1d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680148678035 2023-03-30 03:58:01,565 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting c2d75fd00950431190f2517a5a0fef45, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680148680273 2023-03-30 03:58:01,565 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting acb26dde7b014c018bb90cb7b2866dac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680148680276 2023-03-30 03:58:01,570 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#B#compaction#366 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:58:01,586 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:01,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:01,595 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#A#compaction#367 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:58:01,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:01,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:01,598 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:01,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:01,598 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:01,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,599 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:01,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,600 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:01,600 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:01,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:01,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:01,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:01,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:01,618 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:01,618 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:01,618 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:01,623 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/10c87c21a0e44dde84e8275a68133e55 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/10c87c21a0e44dde84e8275a68133e55 2023-03-30 03:58:01,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:01,626 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,626 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:01,626 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:01,627 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-30 03:58:01,627 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:58:01,627 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:01,627 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:58:01,627 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:01,627 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:58:01,627 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:01,628 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/B of 97a07dc2687acd6de72ca851295ed5b2 into 10c87c21a0e44dde84e8275a68133e55(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:01,628 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:01,628 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/B, priority=13, startTime=1680148681563; duration=0sec 2023-03-30 03:58:01,628 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:01,628 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:01,629 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:01,629 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/C is initiating minor compaction (all files) 2023-03-30 03:58:01,629 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/C in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:01,629 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bc4e8fedb5ba454292961c5e87417335, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e1f62204bce5496f8c45b34c7670a052, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8c433e26307640e29620d93d354240f2] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=36.3 K 2023-03-30 03:58:01,629 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting bc4e8fedb5ba454292961c5e87417335, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680148675689 2023-03-30 03:58:01,629 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e1f62204bce5496f8c45b34c7670a052, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680148678035 2023-03-30 03:58:01,630 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8c433e26307640e29620d93d354240f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680148680276 2023-03-30 03:58:01,643 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#C#compaction#368 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:58:02,077 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330f0b6e9e12262490780c0eb00ccf297ef to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f0b6e9e12262490780c0eb00ccf297ef 2023-03-30 03:58:02,084 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/b6e50e0790bf4db68edd797e0ef78dbe as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/b6e50e0790bf4db68edd797e0ef78dbe 2023-03-30 03:58:02,092 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/C of 97a07dc2687acd6de72ca851295ed5b2 into b6e50e0790bf4db68edd797e0ef78dbe(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:02,092 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:02,092 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/C, priority=13, startTime=1680148681563; duration=0sec 2023-03-30 03:58:02,092 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:02,094 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=663, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/46a2ffa991264091ad0db86ca5d894af 2023-03-30 03:58:02,098 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/1b94a3f6bd934cfeadda9cba9854573e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/1b94a3f6bd934cfeadda9cba9854573e 2023-03-30 03:58:02,104 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/A of 97a07dc2687acd6de72ca851295ed5b2 into 1b94a3f6bd934cfeadda9cba9854573e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:02,104 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:02,105 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/A, priority=13, startTime=1680148681563; duration=0sec 2023-03-30 03:58:02,105 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:02,120 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/f91394c600534d5f8a14ae115c316918 2023-03-30 03:58:02,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:58:02,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:02,528 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/bf3ab5eea5b44ef9abb109e7dc7fcce0 2023-03-30 03:58:02,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,531 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/46a2ffa991264091ad0db86ca5d894af as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/46a2ffa991264091ad0db86ca5d894af 2023-03-30 03:58:02,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34580 deadline: 1680148742529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148742530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148742531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34610 deadline: 1680148742533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34562 deadline: 1680148742535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,541 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/46a2ffa991264091ad0db86ca5d894af, entries=150, sequenceid=663, filesize=25.4 K 2023-03-30 03:58:02,541 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/f91394c600534d5f8a14ae115c316918 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/f91394c600534d5f8a14ae115c316918 2023-03-30 03:58:02,549 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/f91394c600534d5f8a14ae115c316918, entries=150, sequenceid=663, filesize=11.9 K 2023-03-30 03:58:02,549 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/bf3ab5eea5b44ef9abb109e7dc7fcce0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bf3ab5eea5b44ef9abb109e7dc7fcce0 2023-03-30 03:58:02,554 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bf3ab5eea5b44ef9abb109e7dc7fcce0, entries=150, sequenceid=663, filesize=11.9 K 2023-03-30 03:58:02,554 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 97a07dc2687acd6de72ca851295ed5b2 in 927ms, sequenceid=663, compaction requested=false 2023-03-30 03:58:02,554 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:02,555 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:02,555 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:02,555 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:02,555 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,555 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:02,555 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:02,556 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,556 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,556 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,556 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:02,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20247289[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:02,557 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:02,557 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,558 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:02,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:02,558 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:02,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:02,559 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:02,559 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:02,559 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,559 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:02,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:02,562 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:02,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:02,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:02,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:02,562 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:02,563 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,563 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,563 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:02,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:02,564 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,565 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:02,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:02,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:02,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:02,568 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:02,568 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:02,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:02,568 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:02,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:58:02,635 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 03:58:02,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:58:02,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:02,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:58:02,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:02,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:58:02,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:02,668 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:02,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:02,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:02,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:02,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:02,671 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:02,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:02,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:02,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:02,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,672 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:02,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:02,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:02,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148682658, max timestamp=1680148682672Num uniques 859; 2023-03-30 03:58:02,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:02,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148682658, max timestamp=1680148682672Num uniques 859; 2023-03-30 03:58:02,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:02,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:02,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:02,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:02,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148682658, max timestamp=1680148682672Num uniques 859; 2023-03-30 03:58:02,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:02,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:02,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:02,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:02,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:02,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:02,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:02,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:02,674 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:02,674 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:02,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:02,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:02,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:02,675 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,675 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,676 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,676 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:02,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@670eb2cb[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:02,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:02,677 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,677 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,677 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:02,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:02,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:02,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:02,678 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:02,678 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:02,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:02,680 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:02,680 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:02,680 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:02,680 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:02,680 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:02,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:02,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,682 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:02,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:02,686 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:02,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,686 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,686 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:02,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148742714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148742718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,786 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:02,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:02,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:02,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:02,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:02,789 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:02,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:02,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:02,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:02,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,790 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,790 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:02,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:02,790 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,790 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:02,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:02,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:02,791 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:02,791 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:02,791 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:02,791 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:02,791 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:02,792 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:02,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71be7d06[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:02,793 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:02,793 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,794 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,794 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:02,794 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:02,794 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:02,794 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,794 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:02,795 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:02,795 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:02,795 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:02,795 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:02,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:02,797 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:02,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:02,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:02,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:02,797 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:02,799 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,799 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:02,800 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,801 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,806 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:02,806 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:02,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:02,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:02,806 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:02,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:02,806 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:02,806 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,806 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:02,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:02,807 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:02,807 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:02,807 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,807 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,807 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,807 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148742821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:02,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148742827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,907 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:02,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:02,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:02,909 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:02,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:02,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:02,909 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:02,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,910 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:02,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:02,910 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,911 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,911 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:02,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:02,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:02,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:02,912 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:02,912 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:02,912 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:02,912 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:02,912 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:02,913 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,913 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,913 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,913 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:02,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36642938[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:02,915 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:02,915 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,915 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,915 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:02,916 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:02,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:02,916 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:02,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:02,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:02,917 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,917 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:02,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:02,919 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:02,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:02,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:02,919 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:02,919 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:02,920 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:02,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:02,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:02,921 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,921 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:02,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:02,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:02,922 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,923 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:02,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:02,924 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:02,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:02,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:02,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:02,924 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:02,924 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:02,924 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:02,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:03,025 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:03,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:03,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:03,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:03,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:03,027 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:03,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:03,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:03,027 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:03,028 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,028 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:03,029 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:03,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:03,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:03,029 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:03,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:03,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:03,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:03,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:03,030 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:03,030 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:03,030 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:03,030 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:03,030 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:03,031 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:03,031 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. started... 2023-03-30 03:58:03,031 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. as already flushing 2023-03-30 03:58:03,031 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:03,031 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:03,031 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:03,031 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,031 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:03,031 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:03,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,032 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:03,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:03,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:03,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:03,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3557b547[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:03,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:03,034 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:03,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:03,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:03,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:03,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:03,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:03,035 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,035 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:03,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:03,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34560 deadline: 1680148743028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:03,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34592 deadline: 1680148743032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:03,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:03,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:03,038 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:03,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:03,041 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:03,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:03,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:03,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:03,041 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:03,046 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330644c145ee11d4f1b8105a1271d5461ea to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330644c145ee11d4f1b8105a1271d5461ea 2023-03-30 03:58:03,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:03,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,048 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:03,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:03,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:03,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,050 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:03,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:03,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:03,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:03,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,055 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=677, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/f7854a5d45314501a59be2fe341258ea 2023-03-30 03:58:03,056 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:03,056 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:03,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:03,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:03,056 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,056 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:03,056 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:03,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:03,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:03,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:03,056 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:03,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:03,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:03,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:03,057 DEBUG [Thread-2416] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:03,057 DEBUG [Thread-2416] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:03,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:03,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:03,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/c5a43aa2fefb44ed89292af461c883b6 2023-03-30 03:58:03,072 DEBUG [Thread-2417] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35f0f398 to 127.0.0.1:58033 2023-03-30 03:58:03,072 DEBUG [Thread-2417] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:03,073 DEBUG [Thread-2419] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69fd9cc7 to 127.0.0.1:58033 2023-03-30 03:58:03,073 DEBUG [Thread-2419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:03,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/5315f58d9c564771beb7d87a8ce9cde2 2023-03-30 03:58:03,076 DEBUG [Thread-2425] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3546dad6 to 127.0.0.1:58033 2023-03-30 03:58:03,076 DEBUG [Thread-2425] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:03,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/f7854a5d45314501a59be2fe341258ea as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/f7854a5d45314501a59be2fe341258ea 2023-03-30 03:58:03,078 DEBUG [Thread-2423] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52cea5eb to 127.0.0.1:58033 2023-03-30 03:58:03,078 DEBUG [Thread-2423] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:03,080 DEBUG [Thread-2421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54da4907 to 127.0.0.1:58033 2023-03-30 03:58:03,080 DEBUG [Thread-2421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:03,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/f7854a5d45314501a59be2fe341258ea, entries=200, sequenceid=677, filesize=32.3 K 2023-03-30 03:58:03,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/c5a43aa2fefb44ed89292af461c883b6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c5a43aa2fefb44ed89292af461c883b6 2023-03-30 03:58:03,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c5a43aa2fefb44ed89292af461c883b6, entries=150, sequenceid=677, filesize=11.9 K 2023-03-30 03:58:03,084 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/5315f58d9c564771beb7d87a8ce9cde2 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5315f58d9c564771beb7d87a8ce9cde2 2023-03-30 03:58:03,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5315f58d9c564771beb7d87a8ce9cde2, entries=150, sequenceid=677, filesize=11.9 K 2023-03-30 03:58:03,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 97a07dc2687acd6de72ca851295ed5b2 in 452ms, sequenceid=677, compaction requested=true 2023-03-30 03:58:03,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:03,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:03,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:03,087 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:03,087 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:03,087 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:03,088 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:03,088 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:03,088 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/A is initiating minor compaction (all files) 2023-03-30 03:58:03,088 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/B is initiating minor compaction (all files) 2023-03-30 03:58:03,088 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/A in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:03,088 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/B in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:03,088 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/1b94a3f6bd934cfeadda9cba9854573e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/46a2ffa991264091ad0db86ca5d894af, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/f7854a5d45314501a59be2fe341258ea] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=83.8 K 2023-03-30 03:58:03,089 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/10c87c21a0e44dde84e8275a68133e55, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/f91394c600534d5f8a14ae115c316918, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c5a43aa2fefb44ed89292af461c883b6] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=36.4 K 2023-03-30 03:58:03,089 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1b94a3f6bd934cfeadda9cba9854573e, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680148680276 2023-03-30 03:58:03,089 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 10c87c21a0e44dde84e8275a68133e55, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680148680276 2023-03-30 03:58:03,089 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 46a2ffa991264091ad0db86ca5d894af, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1680148680374 2023-03-30 03:58:03,089 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting f91394c600534d5f8a14ae115c316918, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1680148680374 2023-03-30 03:58:03,089 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting f7854a5d45314501a59be2fe341258ea, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680148682524 2023-03-30 03:58:03,089 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting c5a43aa2fefb44ed89292af461c883b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680148682524 2023-03-30 03:58:03,094 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#B#compaction#376 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:58:03,095 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#A#compaction#375 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:58:03,157 DEBUG [Thread-2416] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:03,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:03,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:58:03,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 03:58:03,340 DEBUG [Thread-2408] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28a98302 to 127.0.0.1:58033 2023-03-30 03:58:03,340 DEBUG [Thread-2408] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:03,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:58:03,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:58:03,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:58:03,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:58:03,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:58:03,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:58:03,341 DEBUG [Thread-2410] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39d5c003 to 127.0.0.1:58033 2023-03-30 03:58:03,341 DEBUG [Thread-2410] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:03,501 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/c44ede94921a4d1c80257937bc4067f8 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c44ede94921a4d1c80257937bc4067f8 2023-03-30 03:58:03,507 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/B of 97a07dc2687acd6de72ca851295ed5b2 into c44ede94921a4d1c80257937bc4067f8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:03,507 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:03,507 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/B, priority=13, startTime=1680148683087; duration=0sec 2023-03-30 03:58:03,507 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:03,507 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:03,508 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:03,508 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 97a07dc2687acd6de72ca851295ed5b2/C is initiating minor compaction (all files) 2023-03-30 03:58:03,508 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97a07dc2687acd6de72ca851295ed5b2/C in TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:03,508 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/b6e50e0790bf4db68edd797e0ef78dbe, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bf3ab5eea5b44ef9abb109e7dc7fcce0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5315f58d9c564771beb7d87a8ce9cde2] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp, totalSize=36.4 K 2023-03-30 03:58:03,508 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b6e50e0790bf4db68edd797e0ef78dbe, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1680148680276 2023-03-30 03:58:03,509 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting bf3ab5eea5b44ef9abb109e7dc7fcce0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1680148680374 2023-03-30 03:58:03,509 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5315f58d9c564771beb7d87a8ce9cde2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680148682524 2023-03-30 03:58:03,515 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 97a07dc2687acd6de72ca851295ed5b2#C#compaction#378 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-03-30 03:58:03,750 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033064d0d54f31ca42678f543c8c952c3588 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033064d0d54f31ca42678f543c8c952c3588 2023-03-30 03:58:03,922 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/aa365d0ad04e423a86225829d851a170 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/aa365d0ad04e423a86225829d851a170 2023-03-30 03:58:03,925 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/C of 97a07dc2687acd6de72ca851295ed5b2 into aa365d0ad04e423a86225829d851a170(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:03,925 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:03,925 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/C, priority=13, startTime=1680148683087; duration=0sec 2023-03-30 03:58:03,926 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:04,155 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=715, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/60355deb45834b6a8ef090dc1150caa0 2023-03-30 03:58:04,311 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/a0a55b60f5bf4500b2935783fa5f9e88 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/a0a55b60f5bf4500b2935783fa5f9e88 2023-03-30 03:58:04,314 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97a07dc2687acd6de72ca851295ed5b2/A of 97a07dc2687acd6de72ca851295ed5b2 into a0a55b60f5bf4500b2935783fa5f9e88(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-30 03:58:04,315 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:04,315 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2., storeName=97a07dc2687acd6de72ca851295ed5b2/A, priority=13, startTime=1680148683087; duration=1sec 2023-03-30 03:58:04,315 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:04,546 DEBUG [Thread-2412] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39e3a826 to 127.0.0.1:58033 2023-03-30 03:58:04,546 DEBUG [Thread-2412] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:04,551 DEBUG [Thread-2406] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29738198 to 127.0.0.1:58033 2023-03-30 03:58:04,551 DEBUG [Thread-2406] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:04,561 DEBUG [Thread-2414] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x019e7fff to 127.0.0.1:58033 2023-03-30 03:58:04,561 DEBUG [Thread-2414] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 138 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 111 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 03:58:04,561 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 1548 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 4639 rows 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 1569 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 4707 rows 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 1559 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 4671 rows 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 1551 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 4653 rows 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 1544 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 4632 rows 2023-03-30 03:58:04,562 INFO [Listener at localhost.localdomain/37215] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 03:58:04,562 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7889ece0 to 127.0.0.1:58033 2023-03-30 03:58:04,562 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:04,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=715 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/2a4822f59cad4d458cac77e50613d320 2023-03-30 03:58:04,564 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 03:58:04,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-30 03:58:04,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:04,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 03:58:04,570 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148684570"}]},"ts":"1680148684570"} 2023-03-30 03:58:04,571 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 03:58:04,572 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 03:58:04,573 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, UNASSIGN}] 2023-03-30 03:58:04,574 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, UNASSIGN 2023-03-30 03:58:04,575 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=97a07dc2687acd6de72ca851295ed5b2, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:04,575 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148684575"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148684575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148684575"}]},"ts":"1680148684575"} 2023-03-30 03:58:04,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=715 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/15dfb27a614b403a986f1a7010d26adf 2023-03-30 03:58:04,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:58:04,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/60355deb45834b6a8ef090dc1150caa0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/60355deb45834b6a8ef090dc1150caa0 2023-03-30 03:58:04,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/60355deb45834b6a8ef090dc1150caa0, entries=150, sequenceid=715, filesize=25.4 K 2023-03-30 03:58:04,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/2a4822f59cad4d458cac77e50613d320 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2a4822f59cad4d458cac77e50613d320 2023-03-30 03:58:04,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2a4822f59cad4d458cac77e50613d320, entries=150, sequenceid=715, filesize=11.9 K 2023-03-30 03:58:04,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/15dfb27a614b403a986f1a7010d26adf as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/15dfb27a614b403a986f1a7010d26adf 2023-03-30 03:58:04,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/15dfb27a614b403a986f1a7010d26adf, entries=150, sequenceid=715, filesize=11.9 K 2023-03-30 03:58:04,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=20.13 KB/20610 for 97a07dc2687acd6de72ca851295ed5b2 in 1256ms, sequenceid=715, compaction requested=false 2023-03-30 03:58:04,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:04,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 03:58:04,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 97a07dc2687acd6de72ca851295ed5b2, disabling compactions & flushes 2023-03-30 03:58:04,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. after waiting 0 ms 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:04,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 97a07dc2687acd6de72ca851295ed5b2 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=A 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=B 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97a07dc2687acd6de72ca851295ed5b2, store=C 2023-03-30 03:58:04,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:04,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 03:58:05,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033096097b9ffc8f4ab2b408735e5421146a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033096097b9ffc8f4ab2b408735e5421146a 2023-03-30 03:58:05,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 03:58:05,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=724, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/4a5936709df140cd8964808abc910fff 2023-03-30 03:58:05,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 03:58:05,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/642892f12a084dc9b512717c9deaedf3 2023-03-30 03:58:06,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/30ebddaf219841a28c17acec87bdb3db 2023-03-30 03:58:06,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/A/4a5936709df140cd8964808abc910fff as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/4a5936709df140cd8964808abc910fff 2023-03-30 03:58:06,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/4a5936709df140cd8964808abc910fff, entries=50, sequenceid=724, filesize=11.7 K 2023-03-30 03:58:06,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/B/642892f12a084dc9b512717c9deaedf3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/642892f12a084dc9b512717c9deaedf3 2023-03-30 03:58:06,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/642892f12a084dc9b512717c9deaedf3, entries=50, sequenceid=724, filesize=7.1 K 2023-03-30 03:58:06,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/.tmp/C/30ebddaf219841a28c17acec87bdb3db as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/30ebddaf219841a28c17acec87bdb3db 2023-03-30 03:58:06,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/30ebddaf219841a28c17acec87bdb3db, entries=50, sequenceid=724, filesize=7.1 K 2023-03-30 03:58:06,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 97a07dc2687acd6de72ca851295ed5b2 in 1660ms, sequenceid=724, compaction requested=true 2023-03-30 03:58:06,418 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/45036a4afec4439780c175835ca6dbab, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/2f285c3b4f504ef5ad4811ba30276b89, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e97f33a262f241acbc1d92d04d7f02a2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/740656a63ab047f28b244f8f9c508ef3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/145e81aebdad47a0958e6beb99ef3e96, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/667fbfbf499844fd881066672e98de3f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bbb75f81063f4a6086b1c1cd0b84666b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/490aa8e9d99241e9988760b384e72cec, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/6efb494f8b8245cba8f247aaceec4073, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/58bfd4e277c14181909bb541d0a1f496, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/28d58105dd2d4e1685df1ef9f4694fa8, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/b4c3fd62b5bf4ea0b752bff9ee2d8708, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ff5ee20c9b70497580655dd22c5c06b8, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ae649dba77124c788791adc245565963, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/eded85f1dc3b48e7859e5b7cb6af7b56, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/8e378c2834794c7ab4a9147aca7a0057, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e8970915c96b439bb79f6c1a3e6e04c5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bfe80dcdc2e740d290df07f24154dd5d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/5266cda90b67438bb306ad8620219e60, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/c2d75fd00950431190f2517a5a0fef45, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/1b94a3f6bd934cfeadda9cba9854573e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/46a2ffa991264091ad0db86ca5d894af, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/f7854a5d45314501a59be2fe341258ea] to archive 2023-03-30 03:58:06,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:58:06,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/45036a4afec4439780c175835ca6dbab to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/45036a4afec4439780c175835ca6dbab 2023-03-30 03:58:06,421 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/2f285c3b4f504ef5ad4811ba30276b89 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/2f285c3b4f504ef5ad4811ba30276b89 2023-03-30 03:58:06,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e97f33a262f241acbc1d92d04d7f02a2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e97f33a262f241acbc1d92d04d7f02a2 2023-03-30 03:58:06,423 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/740656a63ab047f28b244f8f9c508ef3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/740656a63ab047f28b244f8f9c508ef3 2023-03-30 03:58:06,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/145e81aebdad47a0958e6beb99ef3e96 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/145e81aebdad47a0958e6beb99ef3e96 2023-03-30 03:58:06,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/667fbfbf499844fd881066672e98de3f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/667fbfbf499844fd881066672e98de3f 2023-03-30 03:58:06,427 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bbb75f81063f4a6086b1c1cd0b84666b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bbb75f81063f4a6086b1c1cd0b84666b 2023-03-30 03:58:06,428 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/490aa8e9d99241e9988760b384e72cec to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/490aa8e9d99241e9988760b384e72cec 2023-03-30 03:58:06,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/6efb494f8b8245cba8f247aaceec4073 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/6efb494f8b8245cba8f247aaceec4073 2023-03-30 03:58:06,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/58bfd4e277c14181909bb541d0a1f496 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/58bfd4e277c14181909bb541d0a1f496 2023-03-30 03:58:06,431 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/28d58105dd2d4e1685df1ef9f4694fa8 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/28d58105dd2d4e1685df1ef9f4694fa8 2023-03-30 03:58:06,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/b4c3fd62b5bf4ea0b752bff9ee2d8708 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/b4c3fd62b5bf4ea0b752bff9ee2d8708 2023-03-30 03:58:06,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ff5ee20c9b70497580655dd22c5c06b8 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ff5ee20c9b70497580655dd22c5c06b8 2023-03-30 03:58:06,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ae649dba77124c788791adc245565963 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/ae649dba77124c788791adc245565963 2023-03-30 03:58:06,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/eded85f1dc3b48e7859e5b7cb6af7b56 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/eded85f1dc3b48e7859e5b7cb6af7b56 2023-03-30 03:58:06,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/8e378c2834794c7ab4a9147aca7a0057 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/8e378c2834794c7ab4a9147aca7a0057 2023-03-30 03:58:06,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e8970915c96b439bb79f6c1a3e6e04c5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/e8970915c96b439bb79f6c1a3e6e04c5 2023-03-30 03:58:06,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bfe80dcdc2e740d290df07f24154dd5d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/bfe80dcdc2e740d290df07f24154dd5d 2023-03-30 03:58:06,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/5266cda90b67438bb306ad8620219e60 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/5266cda90b67438bb306ad8620219e60 2023-03-30 03:58:06,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/c2d75fd00950431190f2517a5a0fef45 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/c2d75fd00950431190f2517a5a0fef45 2023-03-30 03:58:06,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/1b94a3f6bd934cfeadda9cba9854573e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/1b94a3f6bd934cfeadda9cba9854573e 2023-03-30 03:58:06,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/46a2ffa991264091ad0db86ca5d894af to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/46a2ffa991264091ad0db86ca5d894af 2023-03-30 03:58:06,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/f7854a5d45314501a59be2fe341258ea to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/f7854a5d45314501a59be2fe341258ea 2023-03-30 03:58:06,444 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/a7786a90333f4a51aefa6c9f347f8d6b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/8b726bf3fbbf498b88659e955762c333, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/727163f3aa2a43c6bd8592f0965214f7, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c98caaf090194ebb8d65859777989d4c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/6d41ce0d315b49089131960fa5987ce3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/e021c2d717e042aeb52c0e52ead164a2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c27264af13dd43aebe786aa6e43cb730, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/ff1da3f28e404ee2ab15b715b6b2cc87, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/304114c6000c47bfb3a64eb529912be2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/0ed64d72c775471195f53b8d7080094a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2e06508f892f4beb978cdf2dbd8b7916, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1b79b09ba08e48c8a8e88849d78b6bd0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/143ac5bc83bb4d5d8e7067ec131e2000, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c6e11107c7e7489e864c94d2786332b9, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/696fad92a4e44e93ababda46e3e47151, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1c8f9ab6a79a444386922ed4a527d587, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1a46aedbe51743a0bfef192fce8c290f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/5844ffb4eeea481f97743865cdd45a33, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/3376bd11ba8b441db7f6e427bc0bf1d0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/10c87c21a0e44dde84e8275a68133e55, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/acb26dde7b014c018bb90cb7b2866dac, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/f91394c600534d5f8a14ae115c316918, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c5a43aa2fefb44ed89292af461c883b6] to archive 2023-03-30 03:58:06,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:58:06,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/a7786a90333f4a51aefa6c9f347f8d6b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/a7786a90333f4a51aefa6c9f347f8d6b 2023-03-30 03:58:06,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/8b726bf3fbbf498b88659e955762c333 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/8b726bf3fbbf498b88659e955762c333 2023-03-30 03:58:06,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/727163f3aa2a43c6bd8592f0965214f7 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/727163f3aa2a43c6bd8592f0965214f7 2023-03-30 03:58:06,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c98caaf090194ebb8d65859777989d4c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c98caaf090194ebb8d65859777989d4c 2023-03-30 03:58:06,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/6d41ce0d315b49089131960fa5987ce3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/6d41ce0d315b49089131960fa5987ce3 2023-03-30 03:58:06,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/e021c2d717e042aeb52c0e52ead164a2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/e021c2d717e042aeb52c0e52ead164a2 2023-03-30 03:58:06,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c27264af13dd43aebe786aa6e43cb730 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c27264af13dd43aebe786aa6e43cb730 2023-03-30 03:58:06,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/ff1da3f28e404ee2ab15b715b6b2cc87 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/ff1da3f28e404ee2ab15b715b6b2cc87 2023-03-30 03:58:06,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/304114c6000c47bfb3a64eb529912be2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/304114c6000c47bfb3a64eb529912be2 2023-03-30 03:58:06,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/0ed64d72c775471195f53b8d7080094a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/0ed64d72c775471195f53b8d7080094a 2023-03-30 03:58:06,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2e06508f892f4beb978cdf2dbd8b7916 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2e06508f892f4beb978cdf2dbd8b7916 2023-03-30 03:58:06,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1b79b09ba08e48c8a8e88849d78b6bd0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1b79b09ba08e48c8a8e88849d78b6bd0 2023-03-30 03:58:06,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/143ac5bc83bb4d5d8e7067ec131e2000 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/143ac5bc83bb4d5d8e7067ec131e2000 2023-03-30 03:58:06,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c6e11107c7e7489e864c94d2786332b9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c6e11107c7e7489e864c94d2786332b9 2023-03-30 03:58:06,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/696fad92a4e44e93ababda46e3e47151 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/696fad92a4e44e93ababda46e3e47151 2023-03-30 03:58:06,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1c8f9ab6a79a444386922ed4a527d587 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1c8f9ab6a79a444386922ed4a527d587 2023-03-30 03:58:06,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1a46aedbe51743a0bfef192fce8c290f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/1a46aedbe51743a0bfef192fce8c290f 2023-03-30 03:58:06,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/5844ffb4eeea481f97743865cdd45a33 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/5844ffb4eeea481f97743865cdd45a33 2023-03-30 03:58:06,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/3376bd11ba8b441db7f6e427bc0bf1d0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/3376bd11ba8b441db7f6e427bc0bf1d0 2023-03-30 03:58:06,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/10c87c21a0e44dde84e8275a68133e55 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/10c87c21a0e44dde84e8275a68133e55 2023-03-30 03:58:06,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/acb26dde7b014c018bb90cb7b2866dac to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/acb26dde7b014c018bb90cb7b2866dac 2023-03-30 03:58:06,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/f91394c600534d5f8a14ae115c316918 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/f91394c600534d5f8a14ae115c316918 2023-03-30 03:58:06,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c5a43aa2fefb44ed89292af461c883b6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c5a43aa2fefb44ed89292af461c883b6 2023-03-30 03:58:06,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dad9d3273390487d9147ec8bb3d71a1e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3b2fc136bfd243eeb3f2f12f35fafeec, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/2f6dba1dab9249008ae34a273e11a705, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d66ae52f7ee2454a80533ff8f0413a09, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c02679990ca442ad853b54a2a7b25d0b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a19a7ac2261d4cfaa9766ec6134592e8, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d7091aa6f9ee4c03ba75cf5d9910ef99, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5d132486da874c8eafb735c93471ec5f, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3704da99fa5049c0bb1104d5eb2bc32e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c774e26ec715484093a49b4550d58013, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a91cc64d76aa4eb29c329b36726c111c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e6ede1636e4a4bc8933b162b5aa9cb89, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/26c527925cf64f3f81abdaddc8e029e5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dc77442af7b24d7085950aca67879269, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8b114e2315054219867905d559fbb68e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/79692eb843184f05874f7e57718e67fe, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bc4e8fedb5ba454292961c5e87417335, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5a591466b3ee4cc3a5049461ae717715, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e1f62204bce5496f8c45b34c7670a052, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/b6e50e0790bf4db68edd797e0ef78dbe, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8c433e26307640e29620d93d354240f2, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bf3ab5eea5b44ef9abb109e7dc7fcce0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5315f58d9c564771beb7d87a8ce9cde2] to archive 2023-03-30 03:58:06,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:58:06,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dad9d3273390487d9147ec8bb3d71a1e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dad9d3273390487d9147ec8bb3d71a1e 2023-03-30 03:58:06,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3b2fc136bfd243eeb3f2f12f35fafeec to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3b2fc136bfd243eeb3f2f12f35fafeec 2023-03-30 03:58:06,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/2f6dba1dab9249008ae34a273e11a705 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/2f6dba1dab9249008ae34a273e11a705 2023-03-30 03:58:06,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d66ae52f7ee2454a80533ff8f0413a09 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d66ae52f7ee2454a80533ff8f0413a09 2023-03-30 03:58:06,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c02679990ca442ad853b54a2a7b25d0b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c02679990ca442ad853b54a2a7b25d0b 2023-03-30 03:58:06,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a19a7ac2261d4cfaa9766ec6134592e8 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a19a7ac2261d4cfaa9766ec6134592e8 2023-03-30 03:58:06,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d7091aa6f9ee4c03ba75cf5d9910ef99 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/d7091aa6f9ee4c03ba75cf5d9910ef99 2023-03-30 03:58:06,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5d132486da874c8eafb735c93471ec5f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5d132486da874c8eafb735c93471ec5f 2023-03-30 03:58:06,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3704da99fa5049c0bb1104d5eb2bc32e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/3704da99fa5049c0bb1104d5eb2bc32e 2023-03-30 03:58:06,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c774e26ec715484093a49b4550d58013 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/c774e26ec715484093a49b4550d58013 2023-03-30 03:58:06,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a91cc64d76aa4eb29c329b36726c111c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/a91cc64d76aa4eb29c329b36726c111c 2023-03-30 03:58:06,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e6ede1636e4a4bc8933b162b5aa9cb89 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e6ede1636e4a4bc8933b162b5aa9cb89 2023-03-30 03:58:06,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/26c527925cf64f3f81abdaddc8e029e5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/26c527925cf64f3f81abdaddc8e029e5 2023-03-30 03:58:06,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dc77442af7b24d7085950aca67879269 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/dc77442af7b24d7085950aca67879269 2023-03-30 03:58:06,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8b114e2315054219867905d559fbb68e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8b114e2315054219867905d559fbb68e 2023-03-30 03:58:06,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/79692eb843184f05874f7e57718e67fe to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/79692eb843184f05874f7e57718e67fe 2023-03-30 03:58:06,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bc4e8fedb5ba454292961c5e87417335 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bc4e8fedb5ba454292961c5e87417335 2023-03-30 03:58:06,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5a591466b3ee4cc3a5049461ae717715 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5a591466b3ee4cc3a5049461ae717715 2023-03-30 03:58:06,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e1f62204bce5496f8c45b34c7670a052 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/e1f62204bce5496f8c45b34c7670a052 2023-03-30 03:58:06,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/b6e50e0790bf4db68edd797e0ef78dbe to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/b6e50e0790bf4db68edd797e0ef78dbe 2023-03-30 03:58:06,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8c433e26307640e29620d93d354240f2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/8c433e26307640e29620d93d354240f2 2023-03-30 03:58:06,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bf3ab5eea5b44ef9abb109e7dc7fcce0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/bf3ab5eea5b44ef9abb109e7dc7fcce0 2023-03-30 03:58:06,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5315f58d9c564771beb7d87a8ce9cde2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/5315f58d9c564771beb7d87a8ce9cde2 2023-03-30 03:58:06,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/recovered.edits/727.seqid, newMaxSeqId=727, maxSeqId=4 2023-03-30 03:58:06,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2. 2023-03-30 03:58:06,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 97a07dc2687acd6de72ca851295ed5b2: 2023-03-30 03:58:06,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:58:06,501 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=97a07dc2687acd6de72ca851295ed5b2, regionState=CLOSED 2023-03-30 03:58:06,502 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148686501"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148686501"}]},"ts":"1680148686501"} 2023-03-30 03:58:06,504 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-30 03:58:06,504 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 97a07dc2687acd6de72ca851295ed5b2, server=jenkins-hbase20.apache.org,41157,1680148568851 in 1.9260 sec 2023-03-30 03:58:06,506 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-30 03:58:06,506 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=97a07dc2687acd6de72ca851295ed5b2, UNASSIGN in 1.9310 sec 2023-03-30 03:58:06,507 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148686507"}]},"ts":"1680148686507"} 2023-03-30 03:58:06,508 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 03:58:06,509 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 03:58:06,510 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9440 sec 2023-03-30 03:58:06,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 03:58:06,674 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-30 03:58:06,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-30 03:58:06,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:06,677 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:06,677 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:06,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-30 03:58:06,682 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:58:06,685 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/recovered.edits] 2023-03-30 03:58:06,688 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/4a5936709df140cd8964808abc910fff to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/4a5936709df140cd8964808abc910fff 2023-03-30 03:58:06,689 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/60355deb45834b6a8ef090dc1150caa0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/60355deb45834b6a8ef090dc1150caa0 2023-03-30 03:58:06,691 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/a0a55b60f5bf4500b2935783fa5f9e88 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/A/a0a55b60f5bf4500b2935783fa5f9e88 2023-03-30 03:58:06,694 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2a4822f59cad4d458cac77e50613d320 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/2a4822f59cad4d458cac77e50613d320 2023-03-30 03:58:06,695 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/642892f12a084dc9b512717c9deaedf3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/642892f12a084dc9b512717c9deaedf3 2023-03-30 03:58:06,696 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c44ede94921a4d1c80257937bc4067f8 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/B/c44ede94921a4d1c80257937bc4067f8 2023-03-30 03:58:06,698 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/15dfb27a614b403a986f1a7010d26adf to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/15dfb27a614b403a986f1a7010d26adf 2023-03-30 03:58:06,699 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/30ebddaf219841a28c17acec87bdb3db to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/30ebddaf219841a28c17acec87bdb3db 2023-03-30 03:58:06,700 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/aa365d0ad04e423a86225829d851a170 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/C/aa365d0ad04e423a86225829d851a170 2023-03-30 03:58:06,702 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/recovered.edits/727.seqid to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2/recovered.edits/727.seqid 2023-03-30 03:58:06,702 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/97a07dc2687acd6de72ca851295ed5b2 2023-03-30 03:58:06,703 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:58:06,703 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 03:58:06,703 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-30 03:58:06,705 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033012ee0959cbae4e2e979639262c1bf72d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033012ee0959cbae4e2e979639262c1bf72d 2023-03-30 03:58:06,706 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302df676b9b2ab4897b14bbe444c8cb316 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302df676b9b2ab4897b14bbe444c8cb316 2023-03-30 03:58:06,707 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033037960c7610d4458f9de3f3c008031d8e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033037960c7610d4458f9de3f3c008031d8e 2023-03-30 03:58:06,707 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303e38196fd87e414c8428cdb290edc4a3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303e38196fd87e414c8428cdb290edc4a3 2023-03-30 03:58:06,708 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330513e431f20dd4a9dbfc1cd664710ae61 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330513e431f20dd4a9dbfc1cd664710ae61 2023-03-30 03:58:06,709 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330644c145ee11d4f1b8105a1271d5461ea to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330644c145ee11d4f1b8105a1271d5461ea 2023-03-30 03:58:06,709 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033064d0d54f31ca42678f543c8c952c3588 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033064d0d54f31ca42678f543c8c952c3588 2023-03-30 03:58:06,710 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033081df6b1f925d48d287e93ebee05bfc61 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033081df6b1f925d48d287e93ebee05bfc61 2023-03-30 03:58:06,711 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033089c764b6a02040639e9e82db27442e79 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033089c764b6a02040639e9e82db27442e79 2023-03-30 03:58:06,712 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033096097b9ffc8f4ab2b408735e5421146a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033096097b9ffc8f4ab2b408735e5421146a 2023-03-30 03:58:06,713 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c188f6828511487daf508da0280053d6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c188f6828511487daf508da0280053d6 2023-03-30 03:58:06,713 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c9f66efd283942c386dd46f6dcfbd2ad to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c9f66efd283942c386dd46f6dcfbd2ad 2023-03-30 03:58:06,714 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330cb62cec2e3b14e5596ec5d9e0523ddec to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330cb62cec2e3b14e5596ec5d9e0523ddec 2023-03-30 03:58:06,715 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d0b2bf30e8ba486788b8701f201d9eb6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d0b2bf30e8ba486788b8701f201d9eb6 2023-03-30 03:58:06,716 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d56bbc8791924a1fab6edc397484de0a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d56bbc8791924a1fab6edc397484de0a 2023-03-30 03:58:06,717 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d58dde4180e6472eb9bcd7a2cc7ed90c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d58dde4180e6472eb9bcd7a2cc7ed90c 2023-03-30 03:58:06,717 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dd6930aa28cb42f89592e1f535ef2fbc to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dd6930aa28cb42f89592e1f535ef2fbc 2023-03-30 03:58:06,718 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330eb45f9ce9e174a61941d50e81c4118aa to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330eb45f9ce9e174a61941d50e81c4118aa 2023-03-30 03:58:06,719 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f0b6e9e12262490780c0eb00ccf297ef to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f0b6e9e12262490780c0eb00ccf297ef 2023-03-30 03:58:06,720 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f4aeba16768441a5b758693189c8f850 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f4aeba16768441a5b758693189c8f850 2023-03-30 03:58:06,721 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330fd9e1e9204d84287baf3828f45193f99 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330fd9e1e9204d84287baf3828f45193f99 2023-03-30 03:58:06,721 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 03:58:06,722 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:06,724 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 03:58:06,725 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 03:58:06,726 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:06,726 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 03:58:06,726 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680148686726"}]},"ts":"9223372036854775807"} 2023-03-30 03:58:06,728 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 03:58:06,728 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 97a07dc2687acd6de72ca851295ed5b2, NAME => 'TestAcidGuarantees,,1680148659895.97a07dc2687acd6de72ca851295ed5b2.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 03:58:06,728 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 03:58:06,728 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680148686728"}]},"ts":"9223372036854775807"} 2023-03-30 03:58:06,730 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 03:58:06,731 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:06,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-03-30 03:58:06,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-30 03:58:06,778 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-30 03:58:06,790 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=556 (was 554) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=528 (was 491) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=11152 (was 10196) - AvailableMemoryMB LEAK? - 2023-03-30 03:58:06,802 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=556, MaxFileDescriptor=60000, SystemLoadAverage=528, ProcessCount=166, AvailableMemoryMB=11152 2023-03-30 03:58:06,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:58:06,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:58:06,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:06,807 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 03:58:06,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-30 03:58:06,807 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 03:58:06,808 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:06,809 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9 empty. 2023-03-30 03:58:06,809 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:06,809 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:58:06,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 03:58:06,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 03:58:07,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 03:58:07,218 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:58:07,220 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e121dc4bd8e8d27540b9c7067395d6e9, NAME => 'TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp 2023-03-30 03:58:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 03:58:07,632 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:58:07,632 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e121dc4bd8e8d27540b9c7067395d6e9, disabling compactions & flushes 2023-03-30 03:58:07,632 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:07,632 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:07,632 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. after waiting 0 ms 2023-03-30 03:58:07,632 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:07,632 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:07,632 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:07,635 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 03:58:07,636 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148687636"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148687636"}]},"ts":"1680148687636"} 2023-03-30 03:58:07,637 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 03:58:07,638 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 03:58:07,638 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148687638"}]},"ts":"1680148687638"} 2023-03-30 03:58:07,639 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 03:58:07,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e121dc4bd8e8d27540b9c7067395d6e9, ASSIGN}] 2023-03-30 03:58:07,644 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e121dc4bd8e8d27540b9c7067395d6e9, ASSIGN 2023-03-30 03:58:07,645 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e121dc4bd8e8d27540b9c7067395d6e9, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=false 2023-03-30 03:58:07,796 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=e121dc4bd8e8d27540b9c7067395d6e9, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:07,796 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148687796"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148687796"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148687796"}]},"ts":"1680148687796"} 2023-03-30 03:58:07,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:58:07,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 03:58:07,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:07,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e121dc4bd8e8d27540b9c7067395d6e9, NAME => 'TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:58:07,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:58:07,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,953 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,954 DEBUG [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A 2023-03-30 03:58:07,954 DEBUG [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A 2023-03-30 03:58:07,955 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:07,964 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e121dc4bd8e8d27540b9c7067395d6e9 columnFamilyName A 2023-03-30 03:58:07,965 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.HStore(310): Store=e121dc4bd8e8d27540b9c7067395d6e9/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:07,965 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,966 DEBUG [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B 2023-03-30 03:58:07,966 DEBUG [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B 2023-03-30 03:58:07,966 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:07,967 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e121dc4bd8e8d27540b9c7067395d6e9 columnFamilyName B 2023-03-30 03:58:07,967 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.HStore(310): Store=e121dc4bd8e8d27540b9c7067395d6e9/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:07,967 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,968 DEBUG [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C 2023-03-30 03:58:07,968 DEBUG [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C 2023-03-30 03:58:07,969 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:07,969 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e121dc4bd8e8d27540b9c7067395d6e9 columnFamilyName C 2023-03-30 03:58:07,969 INFO [StoreOpener-e121dc4bd8e8d27540b9c7067395d6e9-1] regionserver.HStore(310): Store=e121dc4bd8e8d27540b9c7067395d6e9/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:07,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:07,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:58:07,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:07,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:58:07,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e121dc4bd8e8d27540b9c7067395d6e9; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64854251, jitterRate=-0.033596351742744446}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:58:07,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:07,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., pid=49, masterSystemTime=1680148687949 2023-03-30 03:58:07,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:07,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:07,979 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=e121dc4bd8e8d27540b9c7067395d6e9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:07,979 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148687979"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148687979"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148687979"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148687979"}]},"ts":"1680148687979"} 2023-03-30 03:58:07,983 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-30 03:58:07,983 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 in 184 msec 2023-03-30 03:58:07,984 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-30 03:58:07,984 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e121dc4bd8e8d27540b9c7067395d6e9, ASSIGN in 341 msec 2023-03-30 03:58:07,984 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 03:58:07,984 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148687984"}]},"ts":"1680148687984"} 2023-03-30 03:58:07,985 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 03:58:07,986 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 03:58:07,987 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1830 sec 2023-03-30 03:58:08,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 03:58:08,918 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-30 03:58:08,918 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x1a885842 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:08,938 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9b9721, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:08,940 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:08,943 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39836, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:08,945 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 03:58:08,947 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59766, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 03:58:08,949 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x02e8c6f6 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:08,958 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a7ac4a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:08,959 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x2cb7a788 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:08,969 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@682c0b9e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:08,969 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x41ccedcb to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:08,981 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50d6899d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:08,982 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x40e6f624 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:08,986 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6662e946, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:08,986 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5d7fcf to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:08,997 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d7deb76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:08,998 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x0f4b7807 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:09,006 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d5f04c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:09,006 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x31fc0e3f to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:09,014 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6523d347, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:09,014 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x79db57d4 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:09,023 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28072326, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:09,024 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x0e56ea94 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:09,032 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d90797d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:09,032 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x4486b3c7 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:09,047 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d3df389, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:09,071 DEBUG [hconnection-0x404f6623-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,071 DEBUG [hconnection-0x90b02a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:09,071 DEBUG [hconnection-0x7f890e49-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,072 DEBUG [hconnection-0x111478d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,072 DEBUG [hconnection-0x10eb10eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,074 DEBUG [hconnection-0x26436fcc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,083 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39844, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,083 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39852, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,085 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39876, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,086 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39878, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,085 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39850, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,088 DEBUG [hconnection-0x12fca753-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,088 DEBUG [hconnection-0x78d7616e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,090 DEBUG [hconnection-0x1e081598-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,090 DEBUG [hconnection-0x449bd1f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:09,095 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,098 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39894, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,099 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39884, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,100 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39900, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:09,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:09,104 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39910, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:09,106 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:09,106 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,106 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:09,106 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:09,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,111 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,111 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:09,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:09,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:09,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,116 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:09,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:09,118 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:09,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,118 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:09,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:09,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:58:09,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:09,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:09,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:09,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:09,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:09,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:09,129 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:09,129 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:09,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:09,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:09,129 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:09,131 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,131 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,132 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:09,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:09,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ddc1cc[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:09,134 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:09,134 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,135 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:09,135 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:09,136 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:09,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,136 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:09,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:09,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,137 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:09,137 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:09,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:09,144 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:09,144 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:09,144 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:09,144 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:09,144 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:09,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,148 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,148 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,149 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:09,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:09,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,157 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:09,157 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:09,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:09,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,157 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148749156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148749158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148749158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148749160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148749170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,257 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:09,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:09,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:09,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148749270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148749270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148749273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148749273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:09,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:09,276 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:09,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:09,276 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:09,277 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,277 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,277 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,279 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148749275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:09,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:09,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:09,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:09,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:09,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:09,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:09,310 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:09,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:09,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:09,311 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:09,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:09,311 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,331 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:09,331 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:09,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,332 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:09,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:09,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2589de7c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:09,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:09,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:09,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:09,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:09,335 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:09,336 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:09,336 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,336 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,336 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:09,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:09,339 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:09,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:09,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:09,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:09,339 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:09,340 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,340 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,340 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,340 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,342 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,343 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,343 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,343 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,343 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,343 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,344 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:09,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:09,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,344 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:09,344 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:09,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,444 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:09,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:09,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:09,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:09,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:09,460 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:09,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:09,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:09,461 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,461 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,461 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,461 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,462 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,462 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:09,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:09,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:09,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148749472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148749474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148749473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148749475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:09,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:09,482 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:09,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:09,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148749484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:09,498 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:09,500 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,500 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:09,500 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:09,500 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:09,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:09,501 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,505 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:09,505 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:09,506 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:09,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32331e1c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:09,508 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:09,508 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,509 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,509 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:09,510 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:09,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:09,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:09,512 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:09,513 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:09,513 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,513 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:09,515 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:09,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:09,522 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:09,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:09,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:09,522 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:09,522 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:09,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,534 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,535 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,536 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,540 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,540 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,540 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:09,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:09,540 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,540 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,541 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,541 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,541 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:09,541 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,541 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,541 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,541 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,541 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:09,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/1a103f11ff07467a954369cf18795111 2023-03-30 03:58:09,626 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/4cef3da2db7a4355a7563d32c1598362 2023-03-30 03:58:09,641 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:09,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:09,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:09,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:09,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:09,646 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:09,646 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,646 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:09,646 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:09,647 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,647 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,647 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,647 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:09,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:09,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:09,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:09,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:09,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:09,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:09,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:09,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:09,663 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:09,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a467890[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:09,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:09,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,667 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:09,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:09,668 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,668 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:09,668 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:09,669 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:09,669 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,669 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:09,669 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:09,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:09,671 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:09,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:09,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:09,672 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:09,672 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:09,673 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,673 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,673 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,674 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,676 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,680 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:09,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:09,681 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:09,681 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:09,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:09,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,680 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148749775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148749777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,781 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:09,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148749779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148749780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:09,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:09,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:09,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:09,784 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:09,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:09,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:09,784 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,785 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,785 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,786 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,786 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:09,786 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:09,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:09,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:09,787 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:09,787 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,787 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:09,787 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:09,787 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:09,787 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,787 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:09,788 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:09,788 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:09,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:09,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:09,788 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:09,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,789 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,789 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148749788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:09,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@735c0db5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:09,791 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:09,791 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,791 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,792 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,792 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:09,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:09,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:09,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:09,792 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:09,793 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,793 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:09,793 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:09,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:09,797 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:09,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:09,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:09,797 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:09,797 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:09,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,808 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,809 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,813 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,813 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,813 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,813 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,813 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,814 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,814 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,814 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,814 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:09,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:09,814 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:09,814 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:09,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,814 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,914 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:09,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:09,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:09,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:09,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:09,938 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:09,938 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,938 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:09,938 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:09,939 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,939 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,939 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,939 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:09,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:09,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:09,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:09,954 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:09,954 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:09,954 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:09,958 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:09,958 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:09,958 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,958 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:09,958 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:09,958 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:09,959 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:09,959 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:09,959 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,959 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:09,959 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:09,964 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:09,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@184d579c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:09,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:09,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,970 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:09,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:09,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:09,971 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,971 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:09,973 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:09,973 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,973 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:09,974 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:09,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:09,978 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:09,978 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:09,978 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:09,978 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:09,978 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:09,988 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,989 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,990 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:09,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:09,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:09,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:09,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:09,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:09,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:09,998 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:09,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:09,999 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:09,999 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:10,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/3e419744717545aaa6bb93f90679f5cc 2023-03-30 03:58:10,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/1a103f11ff07467a954369cf18795111 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1a103f11ff07467a954369cf18795111 2023-03-30 03:58:10,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1a103f11ff07467a954369cf18795111, entries=150, sequenceid=13, filesize=11.6 K 2023-03-30 03:58:10,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/4cef3da2db7a4355a7563d32c1598362 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4cef3da2db7a4355a7563d32c1598362 2023-03-30 03:58:10,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4cef3da2db7a4355a7563d32c1598362, entries=150, sequenceid=13, filesize=11.6 K 2023-03-30 03:58:10,076 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/3e419744717545aaa6bb93f90679f5cc as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3e419744717545aaa6bb93f90679f5cc 2023-03-30 03:58:10,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3e419744717545aaa6bb93f90679f5cc, entries=150, sequenceid=13, filesize=11.6 K 2023-03-30 03:58:10,080 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e121dc4bd8e8d27540b9c7067395d6e9 in 954ms, sequenceid=13, compaction requested=false 2023-03-30 03:58:10,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:10,099 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:10,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:10,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:10,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:10,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:10,101 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:10,101 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:10,101 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:10,101 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:10,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:10,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:10,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:10,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,103 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:10,103 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,103 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:10,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:10,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:10,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:10,104 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:10,104 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:10,104 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:10,104 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:10,104 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:10,104 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:10,104 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:10,104 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 03:58:10,104 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:10,104 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:10,104 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:10,104 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:10,104 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:10,105 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:10,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:10,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:10,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148750299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148750299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148750300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148750300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148750302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148750410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148750410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148750410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148750410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148750413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,511 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/388307fb3bce41629d449946496e8260 2023-03-30 03:58:10,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148750612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148750612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148750612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148750613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148750617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148750915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148750916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148750917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148750917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,921 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/7fd0bb40cf1247f28f55145a7d71b245 2023-03-30 03:58:10,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:10,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148750925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,938 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/9e16dbb467a44f4bbad37b2eb8900f18 2023-03-30 03:58:10,942 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/388307fb3bce41629d449946496e8260 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/388307fb3bce41629d449946496e8260 2023-03-30 03:58:10,946 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/388307fb3bce41629d449946496e8260, entries=150, sequenceid=33, filesize=11.6 K 2023-03-30 03:58:10,947 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/7fd0bb40cf1247f28f55145a7d71b245 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/7fd0bb40cf1247f28f55145a7d71b245 2023-03-30 03:58:10,950 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/7fd0bb40cf1247f28f55145a7d71b245, entries=150, sequenceid=33, filesize=11.6 K 2023-03-30 03:58:10,952 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/9e16dbb467a44f4bbad37b2eb8900f18 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9e16dbb467a44f4bbad37b2eb8900f18 2023-03-30 03:58:10,956 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9e16dbb467a44f4bbad37b2eb8900f18, entries=150, sequenceid=33, filesize=11.6 K 2023-03-30 03:58:10,956 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e121dc4bd8e8d27540b9c7067395d6e9 in 852ms, sequenceid=33, compaction requested=false 2023-03-30 03:58:10,957 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:10,957 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:10,957 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:10,957 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:10,957 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,957 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:10,957 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:10,958 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,958 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:10,958 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:10,958 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32a47977[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:10,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:10,960 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:10,960 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:10,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:10,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:10,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:10,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:10,961 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,961 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:10,963 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:10,966 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:10,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:10,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:10,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:10,966 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:10,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:10,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:10,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:10,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:10,973 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,974 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,975 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,975 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:10,975 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:10,975 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:10,975 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:10,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:10,975 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:10,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:10,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:10,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:10,975 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:10,976 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:10,976 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:10,975 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:10,976 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:10,976 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:10,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:11,076 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:11,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:11,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:11,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:11,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:11,078 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:11,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:11,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:11,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:11,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,078 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,079 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,079 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:11,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:11,079 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,079 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:11,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:11,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:11,080 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:11,080 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:11,080 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:11,080 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:11,080 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:11,080 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:11,080 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:11,080 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 03:58:11,080 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:11,080 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:11,080 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:11,080 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:11,080 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:11,080 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:11,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:11,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:11,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148751451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148751451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148751452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148751458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148751463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,487 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/38ac5dd572f9447f880bc3ff27f9d371 2023-03-30 03:58:11,558 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/8aa07406a01a43ad94c6a580b8245e53 2023-03-30 03:58:11,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148751564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148751564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148751565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148751566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148751575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,609 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/1c97eaf6bc5749ab99fc07cdef30fb25 2023-03-30 03:58:11,616 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/38ac5dd572f9447f880bc3ff27f9d371 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/38ac5dd572f9447f880bc3ff27f9d371 2023-03-30 03:58:11,621 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/38ac5dd572f9447f880bc3ff27f9d371, entries=150, sequenceid=45, filesize=11.6 K 2023-03-30 03:58:11,622 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/8aa07406a01a43ad94c6a580b8245e53 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/8aa07406a01a43ad94c6a580b8245e53 2023-03-30 03:58:11,628 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/8aa07406a01a43ad94c6a580b8245e53, entries=150, sequenceid=45, filesize=11.6 K 2023-03-30 03:58:11,629 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/1c97eaf6bc5749ab99fc07cdef30fb25 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1c97eaf6bc5749ab99fc07cdef30fb25 2023-03-30 03:58:11,641 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1c97eaf6bc5749ab99fc07cdef30fb25, entries=150, sequenceid=45, filesize=11.6 K 2023-03-30 03:58:11,642 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e121dc4bd8e8d27540b9c7067395d6e9 in 562ms, sequenceid=45, compaction requested=true 2023-03-30 03:58:11,643 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:11,643 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:11,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:11,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:11,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:11,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:11,645 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,645 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,646 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:11,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b2f154b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:11,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:11,650 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,656 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:11,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:11,656 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:11,657 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,657 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:11,657 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:11,657 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:11,657 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:11,657 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:11,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:11,662 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:11,662 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:11,662 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:11,662 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:11,662 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:11,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:11,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,665 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:11,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:11,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:11,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,674 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,674 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:11,675 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:11,676 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:11,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:11,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:11,776 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:11,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:11,777 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-03-30 03:58:11,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:11,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:11,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:11,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:11,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:11,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:11,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:11,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:11,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148751784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148751787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148751789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:11,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:11,802 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:11,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:11,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:11,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:11,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148751791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148751793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,804 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:11,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,805 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:11,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:11,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:11,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:11,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:11,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:11,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:11,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:11,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:11,814 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:11,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:11,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:11,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:11,814 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:11,815 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:11,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280d963e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:11,817 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:11,817 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:11,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:11,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:11,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:11,817 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,817 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:11,818 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:11,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:11,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:11,820 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:11,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:11,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:11,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:11,820 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:11,821 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,821 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:11,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,821 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,822 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:11,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:11,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:11,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,826 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:11,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:11,826 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:11,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:11,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:11,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:11,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148751899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148751900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148751905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:11,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148751905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,926 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:11,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:11,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:11,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:11,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:11,928 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:11,928 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:11,929 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:11,929 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:11,929 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,929 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,931 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:11,931 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:11,931 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,931 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:11,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:11,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:11,932 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:11,932 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:11,932 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:11,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:11,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:11,933 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:11,933 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:11,933 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:11,933 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:11,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:11,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:11,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:11,933 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:11,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,934 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:11,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0ec514[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:11,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:11,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,936 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,936 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,936 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:11,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:11,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:11,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:11,937 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:11,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:11,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:11,938 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:11,938 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:11,938 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:11,938 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:11,938 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:11,939 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,939 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,939 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:11,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:11,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,941 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:11,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:11,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:11,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:11,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:11,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:11,942 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:11,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:11,942 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:11,942 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:11,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:11,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:11,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:11,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,045 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:12,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:12,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:12,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:12,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:12,048 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:12,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:12,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:12,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,049 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,049 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,049 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:12,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:12,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:12,050 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:12,050 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,050 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:12,050 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:12,050 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:12,050 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,050 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:12,050 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:12,051 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:12,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:12,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:12,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:12,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:12,052 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 03:58:12,053 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:12,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bf060e3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:12,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:12,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,064 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,064 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:12,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:12,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:12,065 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:12,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,066 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,066 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,066 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:12,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:12,067 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:12,067 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:12,067 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:12,067 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:12,067 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:12,068 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,068 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,068 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,069 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,070 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,072 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:12,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:12,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:12,072 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,072 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:12,072 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:12,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148752094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148752102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148752107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148752111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148752111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,173 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:12,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:12,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:12,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:12,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:12,175 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:12,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:12,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:12,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,176 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:12,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:12,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:12,177 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,177 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:12,177 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:12,177 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:12,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:12,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,178 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,179 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,179 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:12,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b6b19b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:12,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:12,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,180 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,180 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:12,180 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:12,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,181 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:12,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:12,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:12,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,181 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:12,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:12,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:12,183 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:12,183 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:12,183 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:12,183 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:12,183 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:12,184 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,184 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,184 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,185 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:12,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:12,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,187 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,187 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:12,216 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/9d45dbdc97364860b0c8ca0756becc3e 2023-03-30 03:58:12,223 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/9f40e9a09d1d4232929494e9dd968d3a 2023-03-30 03:58:12,288 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:12,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:12,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:12,290 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:12,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:12,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:12,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,292 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:12,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:12,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:12,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:12,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:12,293 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,293 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,293 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:12,293 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:12,295 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,295 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,295 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,295 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:12,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cce6ad5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:12,296 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:12,296 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,297 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,297 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:12,297 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,297 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:12,297 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:12,297 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:12,298 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,298 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,298 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,298 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:12,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:12,300 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:12,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:12,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:12,300 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:12,300 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:12,301 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,301 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,301 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,301 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,302 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,303 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:12,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:12,303 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,304 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,304 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,304 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:12,304 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:12,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:12,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,404 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:12,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:12,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:12,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:12,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:12,407 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:12,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:12,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:12,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,408 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:12,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:12,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:12,409 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,409 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:12,409 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:12,409 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:12,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:12,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148752407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:12,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c4303d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:12,413 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:12,414 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148752413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,423 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,423 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:12,423 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,423 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:12,423 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:12,423 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:12,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148752417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148752418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,430 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,430 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:12,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:12,432 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:12,432 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:12,432 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:12,432 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:12,432 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:12,439 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,439 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,439 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,440 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,441 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:12,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:12,447 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,447 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:12,447 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:12,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,547 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:12,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:12,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:12,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:12,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:12,550 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:12,550 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,550 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:12,550 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:12,550 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,550 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:12,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:12,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:12,552 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:12,552 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,552 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:12,552 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:12,552 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:12,552 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,552 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:12,552 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:12,552 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,552 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:12,552 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:12,553 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,553 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:12,553 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:12,553 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,553 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,554 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,554 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:12,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@287ba3be[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:12,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:12,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,555 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:12,555 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:12,555 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:12,556 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,556 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:12,557 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,557 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:12,557 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,557 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:12,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:12,558 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:12,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:12,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:12,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:12,559 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:12,559 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,559 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:12,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,559 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,560 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:12,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:12,561 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:12,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:12,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:12,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:12,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:12,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:12,562 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,563 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:12,563 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:12,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148752601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/977d144c1a1842e7a568812f235f5dbb 2023-03-30 03:58:12,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/9d45dbdc97364860b0c8ca0756becc3e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9d45dbdc97364860b0c8ca0756becc3e 2023-03-30 03:58:12,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9d45dbdc97364860b0c8ca0756becc3e, entries=250, sequenceid=67, filesize=16.2 K 2023-03-30 03:58:12,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/9f40e9a09d1d4232929494e9dd968d3a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9f40e9a09d1d4232929494e9dd968d3a 2023-03-30 03:58:12,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9f40e9a09d1d4232929494e9dd968d3a, entries=150, sequenceid=67, filesize=11.6 K 2023-03-30 03:58:12,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/977d144c1a1842e7a568812f235f5dbb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/977d144c1a1842e7a568812f235f5dbb 2023-03-30 03:58:12,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/977d144c1a1842e7a568812f235f5dbb, entries=150, sequenceid=67, filesize=11.6 K 2023-03-30 03:58:12,643 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=46.96 KB/48090 for e121dc4bd8e8d27540b9c7067395d6e9 in 866ms, sequenceid=67, compaction requested=true 2023-03-30 03:58:12,643 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:12,643 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:12,643 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:58:12,643 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:58:12,643 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:12,643 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:12,644 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:58:12,644 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:58:12,644 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/A is initiating minor compaction (all files) 2023-03-30 03:58:12,644 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/B is initiating minor compaction (all files) 2023-03-30 03:58:12,644 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/A in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,644 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/B in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,645 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1a103f11ff07467a954369cf18795111, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/388307fb3bce41629d449946496e8260, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/38ac5dd572f9447f880bc3ff27f9d371, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9d45dbdc97364860b0c8ca0756becc3e] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=51 K 2023-03-30 03:58:12,645 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4cef3da2db7a4355a7563d32c1598362, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/7fd0bb40cf1247f28f55145a7d71b245, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/8aa07406a01a43ad94c6a580b8245e53, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9f40e9a09d1d4232929494e9dd968d3a] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=46.4 K 2023-03-30 03:58:12,646 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1a103f11ff07467a954369cf18795111, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680148689100 2023-03-30 03:58:12,646 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 4cef3da2db7a4355a7563d32c1598362, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680148689100 2023-03-30 03:58:12,646 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 388307fb3bce41629d449946496e8260, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680148689150 2023-03-30 03:58:12,646 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 7fd0bb40cf1247f28f55145a7d71b245, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680148689150 2023-03-30 03:58:12,647 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 38ac5dd572f9447f880bc3ff27f9d371, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680148690298 2023-03-30 03:58:12,647 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 8aa07406a01a43ad94c6a580b8245e53, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680148690298 2023-03-30 03:58:12,647 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 9d45dbdc97364860b0c8ca0756becc3e, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680148691449 2023-03-30 03:58:12,647 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9f40e9a09d1d4232929494e9dd968d3a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680148691457 2023-03-30 03:58:12,654 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#B#compaction#396 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:12,660 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#A#compaction#397 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:12,663 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:12,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:12,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:12,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:12,670 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:12,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:12,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:12,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,670 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:12,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:12,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:12,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:12,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:12,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:12,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:12,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:12,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:12,694 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:12,694 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:12,694 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,694 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:12,694 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:12,694 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:12,694 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:12,695 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:12,695 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:12,695 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:12,695 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:12,699 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/a1d1c39545424c8da3f23bebddfc6284 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a1d1c39545424c8da3f23bebddfc6284 2023-03-30 03:58:12,708 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/72225116d88b41d7998a2ff905b4f12c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/72225116d88b41d7998a2ff905b4f12c 2023-03-30 03:58:12,710 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/B of e121dc4bd8e8d27540b9c7067395d6e9 into a1d1c39545424c8da3f23bebddfc6284(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:12,710 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:12,710 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/B, priority=12, startTime=1680148692643; duration=0sec 2023-03-30 03:58:12,710 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:12,710 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 03:58:12,711 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 03:58:12,711 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/C is initiating minor compaction (all files) 2023-03-30 03:58:12,711 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/C in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:12,711 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3e419744717545aaa6bb93f90679f5cc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9e16dbb467a44f4bbad37b2eb8900f18, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1c97eaf6bc5749ab99fc07cdef30fb25, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/977d144c1a1842e7a568812f235f5dbb] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=46.4 K 2023-03-30 03:58:12,711 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 3e419744717545aaa6bb93f90679f5cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680148689100 2023-03-30 03:58:12,711 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/A of e121dc4bd8e8d27540b9c7067395d6e9 into 72225116d88b41d7998a2ff905b4f12c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:12,711 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:12,712 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/A, priority=12, startTime=1680148692643; duration=0sec 2023-03-30 03:58:12,712 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:12,712 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9e16dbb467a44f4bbad37b2eb8900f18, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680148689150 2023-03-30 03:58:12,712 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1c97eaf6bc5749ab99fc07cdef30fb25, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680148690298 2023-03-30 03:58:12,712 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 977d144c1a1842e7a568812f235f5dbb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680148691457 2023-03-30 03:58:12,714 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/8d0cd15b3e2342ac9479dd8170497df6 2023-03-30 03:58:12,718 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#C#compaction#399 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:12,731 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/d5e27e4fa7984453a7bdc1fa2678e7a0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d5e27e4fa7984453a7bdc1fa2678e7a0 2023-03-30 03:58:12,731 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/9bfdbd8124874e4e8e71a5840ce826bb 2023-03-30 03:58:12,736 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/C of e121dc4bd8e8d27540b9c7067395d6e9 into d5e27e4fa7984453a7bdc1fa2678e7a0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:12,736 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:12,736 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/C, priority=12, startTime=1680148692643; duration=0sec 2023-03-30 03:58:12,736 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:12,918 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:12,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148692939, max timestamp=1680148692949Num uniques 859; 2023-03-30 03:58:12,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148692939, max timestamp=1680148692949Num uniques 859; 2023-03-30 03:58:12,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:12,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148692939, max timestamp=1680148692949Num uniques 859; 2023-03-30 03:58:12,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:12,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:12,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148752971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148752972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148752974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:12,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:12,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148752977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148753078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148753079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148753079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148753083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,142 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/6d047c339cb34f40a9fbe2ddb207eecb 2023-03-30 03:58:13,145 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/8d0cd15b3e2342ac9479dd8170497df6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8d0cd15b3e2342ac9479dd8170497df6 2023-03-30 03:58:13,148 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8d0cd15b3e2342ac9479dd8170497df6, entries=150, sequenceid=77, filesize=11.6 K 2023-03-30 03:58:13,149 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/9bfdbd8124874e4e8e71a5840ce826bb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9bfdbd8124874e4e8e71a5840ce826bb 2023-03-30 03:58:13,152 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9bfdbd8124874e4e8e71a5840ce826bb, entries=150, sequenceid=77, filesize=11.6 K 2023-03-30 03:58:13,153 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/6d047c339cb34f40a9fbe2ddb207eecb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/6d047c339cb34f40a9fbe2ddb207eecb 2023-03-30 03:58:13,156 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/6d047c339cb34f40a9fbe2ddb207eecb, entries=150, sequenceid=77, filesize=11.6 K 2023-03-30 03:58:13,157 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e121dc4bd8e8d27540b9c7067395d6e9 in 463ms, sequenceid=77, compaction requested=false 2023-03-30 03:58:13,157 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:13,157 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:13,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:13,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:13,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:13,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:13,158 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:13,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:13,158 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:13,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:13,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15fce543[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:13,159 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:13,160 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:13,160 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:13,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:13,160 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,160 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:13,160 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:13,160 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:13,160 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:13,161 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,161 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:13,161 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:13,161 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:13,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:13,163 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:13,163 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:13,163 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:13,163 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:13,163 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:13,164 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,164 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:13,164 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:13,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:13,164 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:13,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:13,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:13,165 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:13,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:13,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:13,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:13,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:13,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:13,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:13,167 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:13,166 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:13,167 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:13,167 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:13,167 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:13,167 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:13,167 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:13,167 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,267 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:13,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:13,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:13,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:13,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:13,269 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:13,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:13,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:13,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:13,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,270 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:13,271 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:13,271 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:13,271 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,271 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:13,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:13,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:13,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:13,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:13,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:13,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:13,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:13,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:13,272 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:13,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:13,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:13,272 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-30 03:58:13,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:13,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:13,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:13,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:13,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:13,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:13,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:13,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:13,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148753311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148753321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148753322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148753322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148753423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148753431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148753433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148753433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148753611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148753628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148753634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148753639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148753639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,679 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/186654ffefe34a329249224092ac998e 2023-03-30 03:58:13,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148753934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148753938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148753945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:13,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148753946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,088 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/fa0af5c83a284f67983264e63ae50b1d 2023-03-30 03:58:14,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:14,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148754441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:14,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148754443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:14,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148754452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:14,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148754454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,499 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/e6e6f43ba2e644eea0a22f84740a6f30 2023-03-30 03:58:14,502 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/186654ffefe34a329249224092ac998e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/186654ffefe34a329249224092ac998e 2023-03-30 03:58:14,504 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/186654ffefe34a329249224092ac998e, entries=150, sequenceid=115, filesize=11.6 K 2023-03-30 03:58:14,505 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/fa0af5c83a284f67983264e63ae50b1d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/fa0af5c83a284f67983264e63ae50b1d 2023-03-30 03:58:14,507 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/fa0af5c83a284f67983264e63ae50b1d, entries=150, sequenceid=115, filesize=11.6 K 2023-03-30 03:58:14,508 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/e6e6f43ba2e644eea0a22f84740a6f30 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/e6e6f43ba2e644eea0a22f84740a6f30 2023-03-30 03:58:14,513 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/e6e6f43ba2e644eea0a22f84740a6f30, entries=150, sequenceid=115, filesize=11.6 K 2023-03-30 03:58:14,513 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=53.67 KB/54960 for e121dc4bd8e8d27540b9c7067395d6e9 in 1241ms, sequenceid=115, compaction requested=true 2023-03-30 03:58:14,513 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:14,514 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:14,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:14,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:14,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:14,514 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:14,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:14,515 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:14,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:14,515 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:14,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:14,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2adbbdeb[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:14,516 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:14,516 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:14,517 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:14,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:14,517 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,517 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:14,518 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:14,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:14,529 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:14,529 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:14,529 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:14,529 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:14,529 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:14,531 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,531 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:14,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,532 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:14,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:14,532 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:14,533 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,534 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,534 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:14,534 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:14,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:14,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:14,534 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:14,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:14,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:14,535 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:14,534 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:14,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:14,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:14,535 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:14,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:14,535 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:14,535 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,635 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:14,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:14,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:14,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:14,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:14,638 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:14,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:14,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:14,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:14,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:14,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,640 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:14,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:14,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:14,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:14,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:14,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:14,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:14,642 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 03:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:15,050 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/5cac787d766448498fb08551a8fa617e 2023-03-30 03:58:15,061 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/5f3d96ede76246759ff12af4df1f946b 2023-03-30 03:58:15,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:15,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:15,470 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/ded112327d644c08ba74100db874ace4 2023-03-30 03:58:15,473 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/5cac787d766448498fb08551a8fa617e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/5cac787d766448498fb08551a8fa617e 2023-03-30 03:58:15,476 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/5cac787d766448498fb08551a8fa617e, entries=150, sequenceid=126, filesize=11.6 K 2023-03-30 03:58:15,477 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/5f3d96ede76246759ff12af4df1f946b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5f3d96ede76246759ff12af4df1f946b 2023-03-30 03:58:15,479 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5f3d96ede76246759ff12af4df1f946b, entries=150, sequenceid=126, filesize=11.6 K 2023-03-30 03:58:15,480 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/ded112327d644c08ba74100db874ace4 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/ded112327d644c08ba74100db874ace4 2023-03-30 03:58:15,483 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/ded112327d644c08ba74100db874ace4, entries=150, sequenceid=126, filesize=11.6 K 2023-03-30 03:58:15,483 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=114.05 KB/116790 for e121dc4bd8e8d27540b9c7067395d6e9 in 841ms, sequenceid=126, compaction requested=true 2023-03-30 03:58:15,483 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:15,483 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:15,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:15,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:15,483 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:15,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,484 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,484 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:15,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b5f3555[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:15,485 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:15,485 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-03-30 03:58:15,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:15,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148695475, max timestamp=1680148695485Num uniques 859; 2023-03-30 03:58:15,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:15,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148695475, max timestamp=1680148695485Num uniques 859; 2023-03-30 03:58:15,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:15,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148695475, max timestamp=1680148695485Num uniques 859; 2023-03-30 03:58:15,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:15,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:15,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,488 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:15,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:15,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:15,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:15,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:15,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:15,488 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,489 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:15,489 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:15,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:15,495 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:15,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:15,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:15,495 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:15,495 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:15,505 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,505 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,505 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,507 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,508 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:15,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:15,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,508 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,508 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:15,508 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:15,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:15,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680148695497, max timestamp=1680148695524Num uniques 859; 2023-03-30 03:58:15,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680148695497, max timestamp=1680148695524Num uniques 859; 2023-03-30 03:58:15,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680148695497, max timestamp=1680148695524Num uniques 859; 2023-03-30 03:58:15,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 03:58:15,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 03:58:15,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 03:58:15,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148755551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148755564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148755565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148755565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,608 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:15,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:15,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:15,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:15,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:15,611 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:15,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:15,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:15,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:15,611 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,612 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,612 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,613 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:15,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:15,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:15,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:15,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:15,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:15,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:15,613 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:15,613 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,613 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:15,614 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:15,614 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:15,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:15,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:15,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:15,614 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,614 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,615 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:15,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4c0dd3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:15,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:15,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,616 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,616 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:15,616 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:15,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:15,616 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:15,617 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:15,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:15,619 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:15,619 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:15,619 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:15,619 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:15,619 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:15,621 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,621 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,621 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,621 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,622 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,624 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:15,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,624 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:15,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,624 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,624 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148755624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,630 DEBUG [Thread-3065] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:15,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148755666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148755673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148755674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148755674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,724 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:15,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:15,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:15,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:15,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:15,727 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:15,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:15,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:15,727 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:15,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,728 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,728 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:15,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:15,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:15,729 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:15,729 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:15,729 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:15,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:15,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:15,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:15,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:15,730 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:15,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:15,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:15,730 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:15,731 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,731 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:15,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c6825a4[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:15,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:15,732 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,733 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,733 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:15,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,733 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:15,733 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:15,733 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:15,734 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,734 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,734 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,734 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:15,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:15,736 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:15,736 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:15,736 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:15,736 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:15,736 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:15,737 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,737 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,737 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,737 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,738 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,739 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,739 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,739 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:15,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:15,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,740 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,740 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:15,740 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:15,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,840 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:15,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:15,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:15,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:15,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:15,843 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:15,843 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:15,843 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:15,843 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:15,843 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,843 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,844 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,844 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:15,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:15,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:15,845 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:15,845 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:15,845 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:15,845 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:15,845 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:15,845 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,845 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:15,845 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:15,845 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,845 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:15,845 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:15,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:15,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:15,846 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,846 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,847 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,847 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:15,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b2be927[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:15,848 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:15,848 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,848 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,848 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:15,848 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:15,848 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:15,848 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,848 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:15,849 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:15,849 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,849 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,849 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:15,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:15,851 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:15,851 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:15,851 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:15,851 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:15,851 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:15,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,852 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,852 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,853 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,855 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:15,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,855 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:15,855 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:15,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148755874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148755881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148755881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:15,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148755881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/e44999edff6441048af54e2fbf5c39af 2023-03-30 03:58:15,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/37b9ca2414a54816bc242a81631285e1 2023-03-30 03:58:15,955 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:15,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:15,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:15,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:15,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:15,958 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:15,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:15,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:15,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:15,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,958 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,959 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,959 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,959 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,959 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:15,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:15,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:15,960 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:15,960 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:15,960 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:15,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:15,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:15,961 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,961 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:15,961 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:15,961 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:15,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:15,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:15,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:15,961 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:15,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,962 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,962 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:15,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@94c928a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:15,963 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:15,963 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,964 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:15,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:15,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,964 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:15,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:15,964 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:15,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:15,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:15,967 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:15,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:15,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:15,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:15,967 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:15,968 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,968 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,968 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:15,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,970 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:15,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:15,972 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:15,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:15,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:15,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:15,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:15,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:15,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:15,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:15,972 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:15,972 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:15,972 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:15,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,073 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:16,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:16,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:16,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:16,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:16,075 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:16,075 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:16,075 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:16,075 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:16,075 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,075 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,076 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:16,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:16,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:16,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:16,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:16,077 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:16,077 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:16,077 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:16,077 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:16,077 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:16,077 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,077 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:16,078 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:16,078 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,078 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,079 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,079 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:16,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:16,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@771024e5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:16,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,080 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,080 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:16,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,080 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:16,080 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:16,081 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:16,082 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,083 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,083 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:16,083 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:16,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:16,086 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:16,086 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:16,086 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:16,086 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:16,086 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:16,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,087 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:16,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:16,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:16,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:16,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:16,090 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:16,090 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:16,090 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:16,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:16,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148756177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,190 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:16,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:16,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:16,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:16,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:16,195 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:16,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:16,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:16,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:16,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,195 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,196 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:16,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:16,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,196 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:16,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:16,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:16,197 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:16,198 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,198 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:16,198 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:16,198 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:16,198 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:16,199 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,199 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,200 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,200 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148756187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148756189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148756190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:16,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af86aa6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:16,203 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:16,203 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,204 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:16,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:16,204 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:16,204 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,204 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:16,205 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:16,205 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:16,205 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:16,205 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:16,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:16,207 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:16,207 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:16,207 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:16,207 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:16,207 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:16,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:16,208 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,210 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,212 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,215 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:16,215 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:16,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:16,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:16,215 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:16,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:16,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:16,216 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:16,216 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:16,215 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,216 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,216 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,216 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,216 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,216 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,216 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:16,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:16,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:16,316 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:16,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/4a37a54b4d7541d3958007408b518102 2023-03-30 03:58:16,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:16,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:16,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:16,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:16,319 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:16,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:16,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:16,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:16,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,319 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/e44999edff6441048af54e2fbf5c39af as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e44999edff6441048af54e2fbf5c39af 2023-03-30 03:58:16,320 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:16,320 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:16,320 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,320 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:16,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:16,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:16,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:16,321 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,321 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:16,321 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:16,321 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:16,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:16,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e44999edff6441048af54e2fbf5c39af, entries=200, sequenceid=149, filesize=14.1 K 2023-03-30 03:58:16,322 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,323 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/37b9ca2414a54816bc242a81631285e1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/37b9ca2414a54816bc242a81631285e1 2023-03-30 03:58:16,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,323 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:16,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:16,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5ec20f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:16,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:16,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,325 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:16,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:16,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:16,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:16,326 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/37b9ca2414a54816bc242a81631285e1, entries=150, sequenceid=149, filesize=11.8 K 2023-03-30 03:58:16,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/4a37a54b4d7541d3958007408b518102 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a37a54b4d7541d3958007408b518102 2023-03-30 03:58:16,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:16,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:16,328 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:16,328 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:16,328 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:16,328 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:16,328 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:16,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a37a54b4d7541d3958007408b518102, entries=150, sequenceid=149, filesize=11.8 K 2023-03-30 03:58:16,329 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=147.60 KB/151140 for e121dc4bd8e8d27540b9c7067395d6e9 in 843ms, sequenceid=149, compaction requested=true 2023-03-30 03:58:16,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:16,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:16,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:16,330 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:58:16,330 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:58:16,330 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,330 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:16,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-30 03:58:16,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,330 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:16,330 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,331 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62108 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:58:16,331 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59718 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:58:16,331 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/A is initiating minor compaction (all files) 2023-03-30 03:58:16,331 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/B is initiating minor compaction (all files) 2023-03-30 03:58:16,331 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/A in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,331 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/B in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,331 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/72225116d88b41d7998a2ff905b4f12c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8d0cd15b3e2342ac9479dd8170497df6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/186654ffefe34a329249224092ac998e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/5cac787d766448498fb08551a8fa617e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e44999edff6441048af54e2fbf5c39af] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=60.7 K 2023-03-30 03:58:16,331 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a1d1c39545424c8da3f23bebddfc6284, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9bfdbd8124874e4e8e71a5840ce826bb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/fa0af5c83a284f67983264e63ae50b1d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5f3d96ede76246759ff12af4df1f946b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/37b9ca2414a54816bc242a81631285e1] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=58.3 K 2023-03-30 03:58:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:16,332 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 72225116d88b41d7998a2ff905b4f12c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680148691457 2023-03-30 03:58:16,332 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting a1d1c39545424c8da3f23bebddfc6284, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680148691457 2023-03-30 03:58:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,332 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8d0cd15b3e2342ac9479dd8170497df6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680148691784 2023-03-30 03:58:16,332 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9bfdbd8124874e4e8e71a5840ce826bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680148691784 2023-03-30 03:58:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,332 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 186654ffefe34a329249224092ac998e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680148692971 2023-03-30 03:58:16,332 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,332 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting fa0af5c83a284f67983264e63ae50b1d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680148692971 2023-03-30 03:58:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:16,333 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5cac787d766448498fb08551a8fa617e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680148693300 2023-03-30 03:58:16,333 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5f3d96ede76246759ff12af4df1f946b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680148693300 2023-03-30 03:58:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:16,333 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e44999edff6441048af54e2fbf5c39af, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680148695481 2023-03-30 03:58:16,333 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 37b9ca2414a54816bc242a81631285e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680148695481 2023-03-30 03:58:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:16,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:16,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:16,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:16,335 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:16,336 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:16,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:16,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:16,341 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#B#compaction#412 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:16,341 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#A#compaction#411 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:16,355 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/8bacd4234dc7449cab20964ebcbbd459 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8bacd4234dc7449cab20964ebcbbd459 2023-03-30 03:58:16,358 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/A of e121dc4bd8e8d27540b9c7067395d6e9 into 8bacd4234dc7449cab20964ebcbbd459(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:16,358 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:16,358 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/A, priority=11, startTime=1680148696330; duration=0sec 2023-03-30 03:58:16,358 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:16,358 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:58:16,360 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59718 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:58:16,360 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/C is initiating minor compaction (all files) 2023-03-30 03:58:16,360 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/C in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,360 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d5e27e4fa7984453a7bdc1fa2678e7a0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/6d047c339cb34f40a9fbe2ddb207eecb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/e6e6f43ba2e644eea0a22f84740a6f30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/ded112327d644c08ba74100db874ace4, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a37a54b4d7541d3958007408b518102] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=58.3 K 2023-03-30 03:58:16,360 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d5e27e4fa7984453a7bdc1fa2678e7a0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680148691457 2023-03-30 03:58:16,360 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 6d047c339cb34f40a9fbe2ddb207eecb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680148691784 2023-03-30 03:58:16,360 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e6e6f43ba2e644eea0a22f84740a6f30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680148692971 2023-03-30 03:58:16,361 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ded112327d644c08ba74100db874ace4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680148693300 2023-03-30 03:58:16,361 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4a37a54b4d7541d3958007408b518102, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680148695481 2023-03-30 03:58:16,367 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#C#compaction#413 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:16,436 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:16,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:16,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:16,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:16,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:16,439 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:16,439 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:16,439 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:16,439 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:16,439 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,439 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,440 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:16,440 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,440 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,440 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:16,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:16,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:16,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:16,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:16,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:16,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:16,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:16,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:16,441 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:16,441 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:16,441 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:16,441 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-03-30 03:58:16,441 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:16,441 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:16,441 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:16,442 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:16,442 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:16,442 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:16,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:16,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:16,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148756708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148756711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148756711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148756713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,756 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/0d519fcb48c14b3aab05ae8f6a6932aa as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/0d519fcb48c14b3aab05ae8f6a6932aa 2023-03-30 03:58:16,759 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/B of e121dc4bd8e8d27540b9c7067395d6e9 into 0d519fcb48c14b3aab05ae8f6a6932aa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:16,759 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:16,759 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/B, priority=11, startTime=1680148696330; duration=0sec 2023-03-30 03:58:16,759 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:16,774 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/fc947bb7563545f28786591fb468655c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/fc947bb7563545f28786591fb468655c 2023-03-30 03:58:16,777 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/C of e121dc4bd8e8d27540b9c7067395d6e9 into fc947bb7563545f28786591fb468655c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:16,777 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:16,777 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/C, priority=11, startTime=1680148696330; duration=0sec 2023-03-30 03:58:16,777 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:16,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148756815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148756819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:16,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148756819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:16,848 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/2901289125be411faedd9ad278f2deb3 2023-03-30 03:58:17,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148757019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148757023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148757024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,257 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/2861f173d02a437fb6c7c9a30e14332c 2023-03-30 03:58:17,266 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/0b22225ddeee44fbb8ad3659b5467c52 2023-03-30 03:58:17,268 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/2901289125be411faedd9ad278f2deb3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/2901289125be411faedd9ad278f2deb3 2023-03-30 03:58:17,271 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/2901289125be411faedd9ad278f2deb3, entries=150, sequenceid=186, filesize=11.8 K 2023-03-30 03:58:17,272 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/2861f173d02a437fb6c7c9a30e14332c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/2861f173d02a437fb6c7c9a30e14332c 2023-03-30 03:58:17,274 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/2861f173d02a437fb6c7c9a30e14332c, entries=150, sequenceid=186, filesize=11.8 K 2023-03-30 03:58:17,275 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/0b22225ddeee44fbb8ad3659b5467c52 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/0b22225ddeee44fbb8ad3659b5467c52 2023-03-30 03:58:17,277 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/0b22225ddeee44fbb8ad3659b5467c52, entries=150, sequenceid=186, filesize=11.8 K 2023-03-30 03:58:17,278 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for e121dc4bd8e8d27540b9c7067395d6e9 in 837ms, sequenceid=186, compaction requested=false 2023-03-30 03:58:17,278 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:17,278 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:17,278 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:17,279 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,279 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,279 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,279 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:17,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73ebdb8c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:17,280 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:17,280 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,281 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,281 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:17,283 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:17,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:17,283 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:17,284 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,284 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:17,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:17,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:17,286 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:17,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:17,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:17,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:17,286 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:17,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,287 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,288 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,288 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:17,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:17,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,290 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:17,290 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:17,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:17,329 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:17,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:17,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:17,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:17,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:17,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:17,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:17,335 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/372a7e7bd1f64b278d103ee0709bfd3b 2023-03-30 03:58:17,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680148697345, max timestamp=1680148697365Num uniques 859; 2023-03-30 03:58:17,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:17,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680148697345, max timestamp=1680148697365Num uniques 859; 2023-03-30 03:58:17,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:17,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680148697345, max timestamp=1680148697365Num uniques 859; 2023-03-30 03:58:17,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:17,390 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:17,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:17,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:17,394 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:17,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:17,394 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:17,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,395 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,395 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,396 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,396 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:17,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:17,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:17,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:17,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:17,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,397 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:17,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:17,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:17,398 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,398 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:17,398 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:17,398 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:17,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:17,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:17,398 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:17,399 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,399 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,399 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,400 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:17,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52aee18[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:17,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:17,402 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,402 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:17,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:17,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:17,402 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,402 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:17,403 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:17,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148757403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:17,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:17,406 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:17,406 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:17,406 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:17,406 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:17,406 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:17,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,410 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:17,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:17,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,410 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:17,410 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:17,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148757404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148757405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:17,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148757506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,510 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:17,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:17,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:17,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:17,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:17,513 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:17,513 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,513 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:17,513 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:17,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,515 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:17,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:17,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:17,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:17,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:17,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:17,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:17,517 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,517 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:17,517 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:17,517 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:17,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:17,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:17,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:17,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148757512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148757513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,518 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:17,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c9e0941[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:17,520 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:17,520 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,520 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:17,520 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,521 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:17,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:17,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:17,521 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:17,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:17,524 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:17,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:17,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:17,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:17,524 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:17,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,529 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:17,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:17,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:17,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,530 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,530 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:17,530 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,630 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:17,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:17,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:17,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:17,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:17,632 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:17,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:17,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:17,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,633 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:17,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:17,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:17,635 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,635 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:17,635 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:17,635 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:17,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:17,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:17,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7438ee67[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:17,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:17,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:17,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:17,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:17,641 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:17,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:17,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:17,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:17,641 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:17,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,645 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,646 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:17,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:17,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,646 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,646 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:17,647 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148757712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148757718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148757719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:17,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148757722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/a5da19a37cee491998337ec14ac47e4e 2023-03-30 03:58:17,747 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:17,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:17,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:17,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:17,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:17,749 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:17,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:17,749 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:17,750 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,750 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,750 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,750 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,751 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,751 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:17,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:17,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:17,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:17,754 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,754 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:17,754 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:17,754 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:17,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:17,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/d02e8b3b5a6d4471bec418c3add54781 2023-03-30 03:58:17,763 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,764 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,764 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:17,765 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/372a7e7bd1f64b278d103ee0709bfd3b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/372a7e7bd1f64b278d103ee0709bfd3b 2023-03-30 03:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:17,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59226cd7[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:17,767 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:17,767 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/372a7e7bd1f64b278d103ee0709bfd3b, entries=150, sequenceid=198, filesize=11.8 K 2023-03-30 03:58:17,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/a5da19a37cee491998337ec14ac47e4e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a5da19a37cee491998337ec14ac47e4e 2023-03-30 03:58:17,772 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,772 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:17,772 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:17,772 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:17,772 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,772 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:17,773 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,773 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:17,773 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,773 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a5da19a37cee491998337ec14ac47e4e, entries=100, sequenceid=198, filesize=9.4 K 2023-03-30 03:58:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/d02e8b3b5a6d4471bec418c3add54781 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d02e8b3b5a6d4471bec418c3add54781 2023-03-30 03:58:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:17,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:17,777 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:17,777 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:17,777 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:17,777 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:17,777 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:17,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d02e8b3b5a6d4471bec418c3add54781, entries=100, sequenceid=198, filesize=9.4 K 2023-03-30 03:58:17,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for e121dc4bd8e8d27540b9c7067395d6e9 in 451ms, sequenceid=198, compaction requested=true 2023-03-30 03:58:17,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:17,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:17,780 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:17,780 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:17,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:17,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:17,781 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36412 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:17,781 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:17,781 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/A is initiating minor compaction (all files) 2023-03-30 03:58:17,781 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/B is initiating minor compaction (all files) 2023-03-30 03:58:17,781 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/A in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,781 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8bacd4234dc7449cab20964ebcbbd459, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/2901289125be411faedd9ad278f2deb3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/372a7e7bd1f64b278d103ee0709bfd3b] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=35.6 K 2023-03-30 03:58:17,781 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/B in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,781 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,781 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:17,781 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 8bacd4234dc7449cab20964ebcbbd459, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680148695481 2023-03-30 03:58:17,781 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,781 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/0d519fcb48c14b3aab05ae8f6a6932aa, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/2861f173d02a437fb6c7c9a30e14332c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a5da19a37cee491998337ec14ac47e4e] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=33.2 K 2023-03-30 03:58:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:17,782 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,782 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 2901289125be411faedd9ad278f2deb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680148695550 2023-03-30 03:58:17,782 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 0d519fcb48c14b3aab05ae8f6a6932aa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680148695481 2023-03-30 03:58:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,782 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 372a7e7bd1f64b278d103ee0709bfd3b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680148696704 2023-03-30 03:58:17,782 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 2861f173d02a437fb6c7c9a30e14332c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680148695550 2023-03-30 03:58:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,783 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a5da19a37cee491998337ec14ac47e4e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680148696704 2023-03-30 03:58:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,783 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,787 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#A#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:17,789 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,789 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,789 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:17,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:17,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:17,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:17,789 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:17,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,790 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:17,790 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:17,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:17,789 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,790 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,790 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,790 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,790 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:17,790 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,795 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#B#compaction#421 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:17,806 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/4f8ebda4f9ad4ec6b9600b6455402a30 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4f8ebda4f9ad4ec6b9600b6455402a30 2023-03-30 03:58:17,809 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/B of e121dc4bd8e8d27540b9c7067395d6e9 into 4f8ebda4f9ad4ec6b9600b6455402a30(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:17,809 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:17,809 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/B, priority=13, startTime=1680148697780; duration=0sec 2023-03-30 03:58:17,810 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:17,810 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:17,810 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:17,810 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/C is initiating minor compaction (all files) 2023-03-30 03:58:17,810 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/C in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,810 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/fc947bb7563545f28786591fb468655c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/0b22225ddeee44fbb8ad3659b5467c52, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d02e8b3b5a6d4471bec418c3add54781] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=33.2 K 2023-03-30 03:58:17,811 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting fc947bb7563545f28786591fb468655c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680148695481 2023-03-30 03:58:17,811 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 0b22225ddeee44fbb8ad3659b5467c52, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680148695550 2023-03-30 03:58:17,811 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d02e8b3b5a6d4471bec418c3add54781, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680148696704 2023-03-30 03:58:17,815 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#C#compaction#422 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:17,829 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/81b82a79554e456088e7fac4cfeafc30 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/81b82a79554e456088e7fac4cfeafc30 2023-03-30 03:58:17,833 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/C of e121dc4bd8e8d27540b9c7067395d6e9 into 81b82a79554e456088e7fac4cfeafc30(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:17,833 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:17,833 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/C, priority=13, startTime=1680148697780; duration=0sec 2023-03-30 03:58:17,833 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:17,890 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:17,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:17,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:17,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:17,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:17,893 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:17,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:17,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:17,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,893 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,894 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:17,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:17,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:17,894 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:17,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:17,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:17,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:17,895 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:17,895 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:17,895 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:17,895 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:17,895 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:17,895 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:17,895 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:17,895 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-30 03:58:17,895 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:17,895 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:17,896 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:17,896 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:17,896 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:17,896 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:18,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:18,021 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:18,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148758029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148758030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148758036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148758137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39894 deadline: 1680148758137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39910 deadline: 1680148758141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,199 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/e1fd21216d2f4e62b62837c3809bc87d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e1fd21216d2f4e62b62837c3809bc87d 2023-03-30 03:58:18,202 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/A of e121dc4bd8e8d27540b9c7067395d6e9 into e1fd21216d2f4e62b62837c3809bc87d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:18,202 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:18,202 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/A, priority=13, startTime=1680148697780; duration=0sec 2023-03-30 03:58:18,202 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:18,303 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/6913e224afc5484198c96b37015d47ab 2023-03-30 03:58:18,313 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/e364fe550cc74834a27dc486b0f47011 2023-03-30 03:58:18,324 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/bde30678aa864d1084554c51b19da734 2023-03-30 03:58:18,327 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/6913e224afc5484198c96b37015d47ab as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/6913e224afc5484198c96b37015d47ab 2023-03-30 03:58:18,330 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/6913e224afc5484198c96b37015d47ab, entries=150, sequenceid=237, filesize=11.8 K 2023-03-30 03:58:18,330 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/e364fe550cc74834a27dc486b0f47011 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e364fe550cc74834a27dc486b0f47011 2023-03-30 03:58:18,333 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e364fe550cc74834a27dc486b0f47011, entries=150, sequenceid=237, filesize=11.8 K 2023-03-30 03:58:18,334 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/bde30678aa864d1084554c51b19da734 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/bde30678aa864d1084554c51b19da734 2023-03-30 03:58:18,336 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/bde30678aa864d1084554c51b19da734, entries=150, sequenceid=237, filesize=11.8 K 2023-03-30 03:58:18,337 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e121dc4bd8e8d27540b9c7067395d6e9 in 442ms, sequenceid=237, compaction requested=false 2023-03-30 03:58:18,337 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:18,337 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,337 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:18,337 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:18,337 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,337 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:18,337 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:18,338 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,338 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:18,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@baa13c5[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:18,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:18,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,340 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:18,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:18,340 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:18,340 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,340 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:18,341 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:18,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,341 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:18,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:18,343 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:18,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:18,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:18,343 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:18,343 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:18,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,344 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,344 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:18,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,345 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:18,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:18,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:18,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:18,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:18,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:18,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:18,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:18,347 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:18,347 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,347 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:18,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,347 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/1e2776a69c13404f92de3758afa9cc83 2023-03-30 03:58:18,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148698370, max timestamp=1680148698403Num uniques 859; 2023-03-30 03:58:18,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:18,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148698370, max timestamp=1680148698403Num uniques 859; 2023-03-30 03:58:18,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148698370, max timestamp=1680148698403Num uniques 859; 2023-03-30 03:58:18,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:18,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:18,447 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:18,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:18,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:18,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:18,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:18,450 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:18,450 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:18,450 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:18,450 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:18,451 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,451 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,452 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:18,452 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:18,452 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,452 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:18,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:18,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:18,453 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,453 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:18,453 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:18,453 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:18,453 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:18,454 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,454 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,454 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,454 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:18,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6712d7f0[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:18,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:18,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,456 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,457 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:18,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:18,457 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:18,457 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:18,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148758450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39844 deadline: 1680148758450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148758450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,458 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,458 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:18,458 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:18,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:18,459 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:18,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:18,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:18,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:18,459 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:18,460 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,460 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:18,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,460 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,461 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:18,464 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:18,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:18,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:18,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,464 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,464 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:18,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:18,464 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:18,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148758558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148758558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148758558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,564 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:18,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:18,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:18,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:18,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:18,567 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:18,567 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:18,567 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:18,567 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:18,567 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,567 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:18,568 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:18,568 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,568 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:18,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:18,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:18,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:18,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:18,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:18,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:18,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:18,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:18,569 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,569 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:18,569 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:18,569 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:18,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:18,569 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,570 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:18,570 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:18,570 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,570 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,571 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,571 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:18,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ea56a3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:18,572 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:18,572 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,572 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,572 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:18,572 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:18,573 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:18,573 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,573 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:18,573 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,573 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,573 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:18,573 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:18,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:18,575 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:18,575 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:18,575 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:18,575 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:18,575 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:18,576 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,576 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:18,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:18,577 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,577 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,578 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,579 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,579 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,579 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,579 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:18,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:18,579 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:18,579 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,579 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:18,580 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:18,580 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,580 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:18,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:18,580 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:18,580 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:18,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:18,680 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:18,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:18,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:18,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:18,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:18,683 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:18,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:18,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:18,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:18,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,684 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,684 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:18,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:18,684 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,684 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:18,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:18,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:18,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:18,685 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:18,686 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,686 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:18,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ee4cc3d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:18,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:18,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,689 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,689 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,689 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:18,689 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:18,689 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:18,689 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:18,690 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:18,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:18,691 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:18,692 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:18,692 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:18,692 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:18,692 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:18,692 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:18,692 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,693 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:18,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:18,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:18,693 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,694 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:18,696 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:18,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:18,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:18,696 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:18,696 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:18,696 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:18,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148758763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148758763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:18,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148758763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/1d3420a70848483ab1bae124c0fe4cc1 2023-03-30 03:58:18,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/857968dcf3d3404288848eb0040260de 2023-03-30 03:58:18,781 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/1e2776a69c13404f92de3758afa9cc83 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1e2776a69c13404f92de3758afa9cc83 2023-03-30 03:58:18,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1e2776a69c13404f92de3758afa9cc83, entries=200, sequenceid=248, filesize=14.1 K 2023-03-30 03:58:18,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/1d3420a70848483ab1bae124c0fe4cc1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/1d3420a70848483ab1bae124c0fe4cc1 2023-03-30 03:58:18,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/1d3420a70848483ab1bae124c0fe4cc1, entries=150, sequenceid=248, filesize=11.8 K 2023-03-30 03:58:18,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/857968dcf3d3404288848eb0040260de as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/857968dcf3d3404288848eb0040260de 2023-03-30 03:58:18,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/857968dcf3d3404288848eb0040260de, entries=150, sequenceid=248, filesize=11.8 K 2023-03-30 03:58:18,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e121dc4bd8e8d27540b9c7067395d6e9 in 447ms, sequenceid=248, compaction requested=true 2023-03-30 03:58:18,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:18,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:18,792 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:18,792 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:18,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:18,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:18,793 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:18,793 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:18,793 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/A is initiating minor compaction (all files) 2023-03-30 03:58:18,793 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/B is initiating minor compaction (all files) 2023-03-30 03:58:18,793 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/A in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,793 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/B in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,793 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e1fd21216d2f4e62b62837c3809bc87d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/6913e224afc5484198c96b37015d47ab, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1e2776a69c13404f92de3758afa9cc83] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=38.0 K 2023-03-30 03:58:18,793 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4f8ebda4f9ad4ec6b9600b6455402a30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e364fe550cc74834a27dc486b0f47011, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/1d3420a70848483ab1bae124c0fe4cc1] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=35.7 K 2023-03-30 03:58:18,793 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting e1fd21216d2f4e62b62837c3809bc87d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680148695550 2023-03-30 03:58:18,793 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4f8ebda4f9ad4ec6b9600b6455402a30, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680148695550 2023-03-30 03:58:18,793 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 6913e224afc5484198c96b37015d47ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680148697394 2023-03-30 03:58:18,793 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e364fe550cc74834a27dc486b0f47011, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680148697394 2023-03-30 03:58:18,794 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1e2776a69c13404f92de3758afa9cc83, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148698020 2023-03-30 03:58:18,794 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 1d3420a70848483ab1bae124c0fe4cc1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148698020 2023-03-30 03:58:18,796 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:18,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:18,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:18,800 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:18,800 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:18,800 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:18,800 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:18,801 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#A#compaction#430 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:18,801 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#B#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:18,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,802 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:18,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:18,813 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:18,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:18,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:18,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:18,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:18,815 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,815 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:18,815 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-30 03:58:18,816 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:18,816 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:18,816 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:18,816 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:18,816 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:18,816 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:18,822 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/a104489199d048039098408140b28e70 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a104489199d048039098408140b28e70 2023-03-30 03:58:18,825 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/B of e121dc4bd8e8d27540b9c7067395d6e9 into a104489199d048039098408140b28e70(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:18,825 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:18,825 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/B, priority=13, startTime=1680148698792; duration=0sec 2023-03-30 03:58:18,825 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:18,825 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:18,826 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:18,826 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/C is initiating minor compaction (all files) 2023-03-30 03:58:18,826 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/C in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:18,826 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/81b82a79554e456088e7fac4cfeafc30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/bde30678aa864d1084554c51b19da734, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/857968dcf3d3404288848eb0040260de] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=35.7 K 2023-03-30 03:58:18,826 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 81b82a79554e456088e7fac4cfeafc30, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680148695550 2023-03-30 03:58:18,827 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting bde30678aa864d1084554c51b19da734, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680148697394 2023-03-30 03:58:18,828 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 857968dcf3d3404288848eb0040260de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148698020 2023-03-30 03:58:18,832 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#C#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:18,841 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/b34c846e4a134c8b887da549d4935092 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b34c846e4a134c8b887da549d4935092 2023-03-30 03:58:18,845 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/C of e121dc4bd8e8d27540b9c7067395d6e9 into b34c846e4a134c8b887da549d4935092(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:18,845 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:18,845 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/C, priority=13, startTime=1680148698792; duration=0sec 2023-03-30 03:58:18,845 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:19,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:19,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:19,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148759073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148759076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148759078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148759179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148759179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148759182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,215 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/f3e5f94a414b4875bd4ba7452a2dc4e5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f3e5f94a414b4875bd4ba7452a2dc4e5 2023-03-30 03:58:19,218 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/A of e121dc4bd8e8d27540b9c7067395d6e9 into f3e5f94a414b4875bd4ba7452a2dc4e5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:19,218 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:19,218 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/A, priority=13, startTime=1680148698792; duration=0sec 2023-03-30 03:58:19,218 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:19,223 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/3801bbc49e284418a4cca0fc8a24e9e6 2023-03-30 03:58:19,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148759381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148759382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148759387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,631 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/5bb7a880e8b04111ba1c7432f22824b3 2023-03-30 03:58:19,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1680148759669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,673 DEBUG [Thread-3065] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8209 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:19,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148759686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148759688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148759691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:19,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148759732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:19,735 DEBUG [Thread-3069] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:20,042 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/d939f59112924083baeb9855e0d1faf1 2023-03-30 03:58:20,045 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/3801bbc49e284418a4cca0fc8a24e9e6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3801bbc49e284418a4cca0fc8a24e9e6 2023-03-30 03:58:20,047 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3801bbc49e284418a4cca0fc8a24e9e6, entries=150, sequenceid=284, filesize=11.9 K 2023-03-30 03:58:20,048 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/5bb7a880e8b04111ba1c7432f22824b3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5bb7a880e8b04111ba1c7432f22824b3 2023-03-30 03:58:20,050 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5bb7a880e8b04111ba1c7432f22824b3, entries=150, sequenceid=284, filesize=11.9 K 2023-03-30 03:58:20,051 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/d939f59112924083baeb9855e0d1faf1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d939f59112924083baeb9855e0d1faf1 2023-03-30 03:58:20,054 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d939f59112924083baeb9855e0d1faf1, entries=150, sequenceid=284, filesize=11.9 K 2023-03-30 03:58:20,054 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e121dc4bd8e8d27540b9c7067395d6e9 in 1239ms, sequenceid=284, compaction requested=false 2023-03-30 03:58:20,054 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:20,055 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:20,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:20,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:20,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:20,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:20,056 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:20,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:20,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:20,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:20,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@450bc01f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:20,057 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:20,057 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:20,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:20,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:20,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:20,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:20,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:20,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:20,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:20,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:20,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:20,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:20,061 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:20,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:20,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:20,061 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:20,061 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:20,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:20,063 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:20,063 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:20,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:20,065 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:20,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:20,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:20,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:20,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:20,065 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:20,065 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:20,165 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:20,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:20,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:20,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:20,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:20,168 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:20,168 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:20,168 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:20,168 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:20,168 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,168 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:20,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,169 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:20,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:20,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:20,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:20,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:20,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:20,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:20,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:20,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:20,170 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:20,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:20,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:20,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:20,170 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:20,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:20,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:20,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:20,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:20,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148700204, max timestamp=1680148700234Num uniques 859; 2023-03-30 03:58:20,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148700204, max timestamp=1680148700234Num uniques 859; 2023-03-30 03:58:20,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:20,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:20,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680148700204, max timestamp=1680148700234Num uniques 859; 2023-03-30 03:58:20,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:20,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148760267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148760268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148760275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148760376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148760376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148760385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,588 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/686e0cd2559f468aabf630b32ba177df 2023-03-30 03:58:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148760586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148760586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148760591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148760893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148760893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:20,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148760899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:20,996 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/41d3d44e6d364b9ab32161af9a1fb051 2023-03-30 03:58:21,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:21,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148761400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:21,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148761400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,404 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/8a8127d1e6574e06aa980a0f5a0fa5bf 2023-03-30 03:58:21,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:21,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148761401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,407 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/686e0cd2559f468aabf630b32ba177df as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/686e0cd2559f468aabf630b32ba177df 2023-03-30 03:58:21,419 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/686e0cd2559f468aabf630b32ba177df, entries=150, sequenceid=297, filesize=11.9 K 2023-03-30 03:58:21,420 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/41d3d44e6d364b9ab32161af9a1fb051 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/41d3d44e6d364b9ab32161af9a1fb051 2023-03-30 03:58:21,424 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/41d3d44e6d364b9ab32161af9a1fb051, entries=150, sequenceid=297, filesize=11.9 K 2023-03-30 03:58:21,425 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/8a8127d1e6574e06aa980a0f5a0fa5bf as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/8a8127d1e6574e06aa980a0f5a0fa5bf 2023-03-30 03:58:21,431 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/8a8127d1e6574e06aa980a0f5a0fa5bf, entries=150, sequenceid=297, filesize=11.9 K 2023-03-30 03:58:21,432 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e121dc4bd8e8d27540b9c7067395d6e9 in 1262ms, sequenceid=297, compaction requested=true 2023-03-30 03:58:21,432 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:21,432 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:21,435 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,435 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:21,435 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:21,435 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:21,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:21,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d72219[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:21,436 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:21,436 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:21,437 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:21,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:21,437 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,437 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:21,438 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:21,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:21,438 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:21,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:21,441 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:21,441 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:21,441 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:21,441 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:21,441 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:21,447 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,448 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:21,448 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,448 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:21,449 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:21,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:21,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:21,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:21,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:21,455 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:21,455 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:21,455 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:21,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:21,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:21,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:21,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:21,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:21,456 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:21,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:21,555 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:21,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:21,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:21,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:21,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:21,558 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:21,558 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:21,558 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:21,558 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:21,558 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,558 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:21,559 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:21,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:21,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:21,559 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:21,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:21,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:21,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:21,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:21,561 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:21,561 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:21,561 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:21,561 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:21,561 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:21,562 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-30 03:58:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:21,968 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/d87cf86f143b42c0bfffbc3f8e31bb2e 2023-03-30 03:58:22,390 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/82f741d693454446bc23ddf83bcb03be 2023-03-30 03:58:22,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:22,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:22,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148762421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148762423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148762424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148762528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148762530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148762531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148762733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148762733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:22,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148762737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,797 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/31a7580654434a4e9500288eb18c8190 2023-03-30 03:58:22,800 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/d87cf86f143b42c0bfffbc3f8e31bb2e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/d87cf86f143b42c0bfffbc3f8e31bb2e 2023-03-30 03:58:22,807 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/d87cf86f143b42c0bfffbc3f8e31bb2e, entries=150, sequenceid=332, filesize=11.9 K 2023-03-30 03:58:22,808 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/82f741d693454446bc23ddf83bcb03be as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/82f741d693454446bc23ddf83bcb03be 2023-03-30 03:58:22,811 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/82f741d693454446bc23ddf83bcb03be, entries=150, sequenceid=332, filesize=11.9 K 2023-03-30 03:58:22,812 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/31a7580654434a4e9500288eb18c8190 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/31a7580654434a4e9500288eb18c8190 2023-03-30 03:58:22,815 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/31a7580654434a4e9500288eb18c8190, entries=150, sequenceid=332, filesize=11.9 K 2023-03-30 03:58:22,815 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for e121dc4bd8e8d27540b9c7067395d6e9 in 1253ms, sequenceid=332, compaction requested=true 2023-03-30 03:58:22,815 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:22,816 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:22,817 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:22,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:22,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@328e256e[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:22,818 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:22,818 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:22,818 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:22,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:22,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:22,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:22,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:22,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:22,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:22,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:22,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,820 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:22,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:22,825 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:22,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:22,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:22,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:22,825 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:22,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:22,826 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:22,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:22,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:22,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:22,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:22,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:22,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:22,829 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:22,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:22,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:22,829 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:22,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:22,830 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:22,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:22,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:22,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:22,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:22,930 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:22,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:22,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:22,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:22,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:22,932 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:22,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:22,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:22,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:22,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:22,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,933 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:22,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:22,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:22,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:22,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:22,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:22,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:22,935 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:22,935 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:22,935 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:22,935 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:22,935 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:22,935 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:22,935 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:22,935 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:22,935 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:23,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:23,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:23,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148703063, max timestamp=1680148703077Num uniques 859; 2023-03-30 03:58:23,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:23,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148703063, max timestamp=1680148703077Num uniques 859; 2023-03-30 03:58:23,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:23,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148703063, max timestamp=1680148703077Num uniques 859; 2023-03-30 03:58:23,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:23,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148763101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148763107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148763110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148763211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148763214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148763222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,341 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/3a489a669db34f24a59c6f02492a55cb 2023-03-30 03:58:23,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148763416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148763420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148763428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148763724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148763724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148763731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,749 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/ab6bbaca3e75478881b8dc7f843938ee 2023-03-30 03:58:23,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:23,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39878 deadline: 1680148763771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:23,777 DEBUG [Thread-3069] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8212 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:24,157 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/b2053fa1895c41a188bcedc1402cb81a 2023-03-30 03:58:24,160 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/3a489a669db34f24a59c6f02492a55cb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3a489a669db34f24a59c6f02492a55cb 2023-03-30 03:58:24,162 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3a489a669db34f24a59c6f02492a55cb, entries=150, sequenceid=342, filesize=11.9 K 2023-03-30 03:58:24,163 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/ab6bbaca3e75478881b8dc7f843938ee as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/ab6bbaca3e75478881b8dc7f843938ee 2023-03-30 03:58:24,166 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/ab6bbaca3e75478881b8dc7f843938ee, entries=150, sequenceid=342, filesize=11.9 K 2023-03-30 03:58:24,166 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/b2053fa1895c41a188bcedc1402cb81a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b2053fa1895c41a188bcedc1402cb81a 2023-03-30 03:58:24,173 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b2053fa1895c41a188bcedc1402cb81a, entries=150, sequenceid=342, filesize=11.9 K 2023-03-30 03:58:24,173 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e121dc4bd8e8d27540b9c7067395d6e9 in 1238ms, sequenceid=342, compaction requested=true 2023-03-30 03:58:24,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:24,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:24,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:24,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:24,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:24,174 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,174 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,174 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:24,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a339226[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:24,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:24,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,176 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,176 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:24,176 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:24,176 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:24,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,176 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:24,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:24,177 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,177 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:24,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:24,178 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:24,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:24,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:24,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:24,178 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:24,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,179 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,182 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:24,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:24,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:24,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,183 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:24,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,183 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,183 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:24,235 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-30 03:58:24,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:24,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:24,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:24,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:24,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:24,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:24,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148764246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148764250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148764257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,283 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:24,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:24,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:24,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:24,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:24,285 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:24,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:24,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:24,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,286 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:24,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:24,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:24,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:24,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:24,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:24,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:24,288 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,288 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:24,288 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:24,288 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:24,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:24,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:24,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:24,289 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,289 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,289 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,289 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:24,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11a3b047[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:24,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:24,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:24,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:24,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:24,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,291 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:24,291 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,292 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,292 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:24,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:24,293 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:24,293 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:24,293 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:24,293 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:24,293 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:24,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,294 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,295 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,297 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,297 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,297 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,298 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,298 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:24,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,298 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,298 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:24,298 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:24,298 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,298 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,298 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148764358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148764360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148764365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,398 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:24,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:24,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:24,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:24,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:24,401 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:24,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:24,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:24,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,402 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,402 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,402 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,402 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:24,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:24,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:24,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:24,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:24,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:24,404 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,404 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,404 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,404 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:24,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c699f80[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:24,405 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:24,405 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,406 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:24,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:24,406 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,406 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:24,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:24,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:24,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:24,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:24,409 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:24,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:24,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:24,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:24,409 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:24,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,410 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,411 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:24,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:24,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:24,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,413 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:24,413 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,413 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,413 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,413 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,413 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,513 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:24,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:24,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:24,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:24,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:24,516 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:24,516 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,516 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:24,516 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:24,516 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,516 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,517 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,517 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,517 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,517 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:24,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:24,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:24,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:24,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:24,518 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:24,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:24,519 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,519 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:24,521 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:24,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0f6c36[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:24,521 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,521 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:24,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:24,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:24,521 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,521 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:24,522 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,522 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:24,522 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,522 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:24,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:24,524 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:24,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:24,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:24,524 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:24,524 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:24,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,525 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:24,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:24,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,528 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,528 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:24,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:24,529 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:24,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,529 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148764565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148764565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148764572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,629 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:24,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:24,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:24,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:24,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:24,631 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:24,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:24,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:24,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/e931ef5e51874a90aa241b21cb7f6d76 2023-03-30 03:58:24,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/3285844dfb8c4c429427543e3ea7ea47 2023-03-30 03:58:24,700 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,700 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:24,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:24,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:24,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:24,701 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:24,701 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,701 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:24,701 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:24,701 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:24,701 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,702 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:24,702 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:24,702 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:24,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:24,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:24,702 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:24,703 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,703 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:24,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49bb7753[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:24,704 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:24,704 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,705 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,705 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:24,705 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:24,705 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:24,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,705 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:24,706 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,706 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:24,706 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,706 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:24,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:24,708 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:24,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:24,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:24,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:24,708 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:24,709 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,709 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,709 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,710 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:24,712 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,712 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:24,712 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:24,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,712 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,812 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:24,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:24,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:24,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:24,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:24,815 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:24,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:24,815 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:24,816 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,816 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,816 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,816 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,816 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,817 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:24,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:24,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:24,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:24,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,817 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:24,819 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,819 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:24,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4596c7ab[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:24,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:24,820 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,821 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:24,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:24,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,821 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:24,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:24,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:24,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:24,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:24,824 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:24,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:24,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:24,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:24,824 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:24,824 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,824 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,824 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,826 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:24,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:24,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,828 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:24,828 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:24,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:24,834 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,834 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148764872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148764872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:24,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148764879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,928 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:24,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:24,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:24,931 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:24,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:24,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:24,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,932 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,932 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,933 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:24,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:24,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:24,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:24,934 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,934 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:24,934 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:24,934 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:24,934 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:24,935 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,935 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,935 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,935 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:24,936 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:24,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1553d506[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:24,936 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,937 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:24,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:24,937 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:24,937 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,937 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:24,938 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,938 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,938 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,938 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:24,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:24,940 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:24,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:24,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:24,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:24,940 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:24,941 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,941 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:24,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,941 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:24,941 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,942 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:24,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:24,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:24,945 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:24,945 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:24,945 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:24,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:24,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:25,045 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:25,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:25,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:25,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:25,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:25,048 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:25,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:25,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:25,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:25,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:25,050 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,050 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:25,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:25,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:25,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:25,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:25,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:25,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:25,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:25,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:25,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:25,051 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:25,051 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:25,051 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:25,051 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:25,051 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:25,052 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:25,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:25,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58eb38a0[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:25,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:25,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:25,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:25,055 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:25,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:25,055 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,055 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:25,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:25,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:25,055 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:25,056 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:25,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:25,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:25,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:25,058 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:25,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:25,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:25,058 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:25,058 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:25,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:25,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:25,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:25,059 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,060 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/4a21ac45139a441d8c8c6e0050a9d5a0 2023-03-30 03:58:25,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:25,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:25,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:25,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:25,062 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:25,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:25,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:25,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,062 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:25,063 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:25,063 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:25,063 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:25,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:25,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:25,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/e931ef5e51874a90aa241b21cb7f6d76 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e931ef5e51874a90aa241b21cb7f6d76 2023-03-30 03:58:25,067 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e931ef5e51874a90aa241b21cb7f6d76, entries=200, sequenceid=379, filesize=14.3 K 2023-03-30 03:58:25,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/3285844dfb8c4c429427543e3ea7ea47 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/3285844dfb8c4c429427543e3ea7ea47 2023-03-30 03:58:25,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/3285844dfb8c4c429427543e3ea7ea47, entries=150, sequenceid=379, filesize=11.9 K 2023-03-30 03:58:25,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/4a21ac45139a441d8c8c6e0050a9d5a0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a21ac45139a441d8c8c6e0050a9d5a0 2023-03-30 03:58:25,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a21ac45139a441d8c8c6e0050a9d5a0, entries=150, sequenceid=379, filesize=11.9 K 2023-03-30 03:58:25,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e121dc4bd8e8d27540b9c7067395d6e9 in 839ms, sequenceid=379, compaction requested=true 2023-03-30 03:58:25,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:25,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:25,074 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 03:58:25,074 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 03:58:25,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:25,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:25,077 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 75918 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 03:58:25,077 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73478 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 03:58:25,077 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/A is initiating minor compaction (all files) 2023-03-30 03:58:25,077 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/B is initiating minor compaction (all files) 2023-03-30 03:58:25,077 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/A in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:25,077 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/B in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:25,077 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a104489199d048039098408140b28e70, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5bb7a880e8b04111ba1c7432f22824b3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/41d3d44e6d364b9ab32161af9a1fb051, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/82f741d693454446bc23ddf83bcb03be, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/ab6bbaca3e75478881b8dc7f843938ee, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/3285844dfb8c4c429427543e3ea7ea47] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=71.8 K 2023-03-30 03:58:25,077 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f3e5f94a414b4875bd4ba7452a2dc4e5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3801bbc49e284418a4cca0fc8a24e9e6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/686e0cd2559f468aabf630b32ba177df, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/d87cf86f143b42c0bfffbc3f8e31bb2e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3a489a669db34f24a59c6f02492a55cb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e931ef5e51874a90aa241b21cb7f6d76] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=74.1 K 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting a104489199d048039098408140b28e70, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148698020 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting f3e5f94a414b4875bd4ba7452a2dc4e5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148698020 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5bb7a880e8b04111ba1c7432f22824b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680148698426 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 3801bbc49e284418a4cca0fc8a24e9e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680148698426 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 41d3d44e6d364b9ab32161af9a1fb051, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680148699072 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 686e0cd2559f468aabf630b32ba177df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680148699072 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 82f741d693454446bc23ddf83bcb03be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680148700249 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting d87cf86f143b42c0bfffbc3f8e31bb2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680148700249 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting ab6bbaca3e75478881b8dc7f843938ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680148702416 2023-03-30 03:58:25,078 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 3a489a669db34f24a59c6f02492a55cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680148702416 2023-03-30 03:58:25,079 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 3285844dfb8c4c429427543e3ea7ea47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680148703099 2023-03-30 03:58:25,079 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting e931ef5e51874a90aa241b21cb7f6d76, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680148703084 2023-03-30 03:58:25,088 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#B#compaction#447 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:25,088 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#A#compaction#448 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:25,099 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/4c9275a7e9dd4601af4397cb555cec9a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4c9275a7e9dd4601af4397cb555cec9a 2023-03-30 03:58:25,102 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/B of e121dc4bd8e8d27540b9c7067395d6e9 into 4c9275a7e9dd4601af4397cb555cec9a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:25,102 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:25,102 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/B, priority=10, startTime=1680148705074; duration=0sec 2023-03-30 03:58:25,102 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:25,102 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 03:58:25,104 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73478 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 03:58:25,104 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/C is initiating minor compaction (all files) 2023-03-30 03:58:25,104 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/C in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:25,104 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b34c846e4a134c8b887da549d4935092, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d939f59112924083baeb9855e0d1faf1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/8a8127d1e6574e06aa980a0f5a0fa5bf, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/31a7580654434a4e9500288eb18c8190, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b2053fa1895c41a188bcedc1402cb81a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a21ac45139a441d8c8c6e0050a9d5a0] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=71.8 K 2023-03-30 03:58:25,104 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b34c846e4a134c8b887da549d4935092, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680148698020 2023-03-30 03:58:25,105 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d939f59112924083baeb9855e0d1faf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680148698426 2023-03-30 03:58:25,105 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 8a8127d1e6574e06aa980a0f5a0fa5bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680148699072 2023-03-30 03:58:25,105 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 31a7580654434a4e9500288eb18c8190, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680148700249 2023-03-30 03:58:25,105 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting b2053fa1895c41a188bcedc1402cb81a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680148702416 2023-03-30 03:58:25,106 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4a21ac45139a441d8c8c6e0050a9d5a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680148703099 2023-03-30 03:58:25,112 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#C#compaction#449 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:25,163 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:25,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:25,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:25,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:25,165 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:25,165 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:25,165 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:25,165 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:25,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,166 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:25,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:25,170 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:25,171 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,171 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:25,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:25,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:25,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:25,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:25,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:25,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:25,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:25,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:25,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:25,172 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:25,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:25,173 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-30 03:58:25,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:25,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:25,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:25,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:25,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:25,173 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:25,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:25,380 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:25,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148705406, max timestamp=1680148705427Num uniques 859; 2023-03-30 03:58:25,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148705406, max timestamp=1680148705427Num uniques 859; 2023-03-30 03:58:25,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:25,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:25,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148705406, max timestamp=1680148705427Num uniques 859; 2023-03-30 03:58:25,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:25,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148765458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148765461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148765462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,496 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/1881f37e8a284de9af5894621776e312 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1881f37e8a284de9af5894621776e312 2023-03-30 03:58:25,499 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/A of e121dc4bd8e8d27540b9c7067395d6e9 into 1881f37e8a284de9af5894621776e312(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:25,499 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:25,499 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/A, priority=10, startTime=1680148705074; duration=0sec 2023-03-30 03:58:25,499 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:25,526 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/15e291603bd4477c9005b3057d480952 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/15e291603bd4477c9005b3057d480952 2023-03-30 03:58:25,530 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/C of e121dc4bd8e8d27540b9c7067395d6e9 into 15e291603bd4477c9005b3057d480952(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:25,530 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:25,530 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/C, priority=10, startTime=1680148705074; duration=0sec 2023-03-30 03:58:25,530 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:25,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148765563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148765568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148765568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,579 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/9ffda31578e345c995adc0c5b3644f26 2023-03-30 03:58:25,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148765767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148765772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:25,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148765772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:25,987 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/e18914a944a244deb18a9b41a0865bba 2023-03-30 03:58:26,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148766074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148766078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148766080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,394 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/5822736b129f446bad8cf9e9a9872730 2023-03-30 03:58:26,397 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/9ffda31578e345c995adc0c5b3644f26 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9ffda31578e345c995adc0c5b3644f26 2023-03-30 03:58:26,400 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9ffda31578e345c995adc0c5b3644f26, entries=150, sequenceid=389, filesize=11.9 K 2023-03-30 03:58:26,400 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/e18914a944a244deb18a9b41a0865bba as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e18914a944a244deb18a9b41a0865bba 2023-03-30 03:58:26,403 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e18914a944a244deb18a9b41a0865bba, entries=150, sequenceid=389, filesize=11.9 K 2023-03-30 03:58:26,403 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/5822736b129f446bad8cf9e9a9872730 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5822736b129f446bad8cf9e9a9872730 2023-03-30 03:58:26,406 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5822736b129f446bad8cf9e9a9872730, entries=150, sequenceid=389, filesize=11.9 K 2023-03-30 03:58:26,406 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for e121dc4bd8e8d27540b9c7067395d6e9 in 1233ms, sequenceid=389, compaction requested=false 2023-03-30 03:58:26,406 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:26,406 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:26,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:26,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:26,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:26,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:26,407 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:26,407 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:26,408 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:26,408 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:26,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:26,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31997546[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:26,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:26,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:26,409 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:26,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:26,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:26,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:26,409 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:26,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:26,410 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:26,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:26,410 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:26,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:26,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:26,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:26,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:26,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:26,412 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:26,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:26,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:26,412 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:26,412 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:26,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:26,412 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:26,413 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:26,414 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:26,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:26,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:26,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:26,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:26,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:26,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:26,415 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:26,415 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:26,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:26,415 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,415 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:26,416 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:26,416 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:26,416 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:26,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:26,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:26,516 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:26,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:26,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:26,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:26,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:26,518 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:26,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:26,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:26,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:26,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,518 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:26,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,519 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:26,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:26,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:26,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:26,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:26,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:26,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:26,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:26,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:26,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:26,521 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-30 03:58:26,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:26,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:26,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:26,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:26,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:26,521 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:26,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:26,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:26,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148766588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148766591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148766592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148766694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148766697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148766697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148766899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148766901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:26,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148766901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:26,927 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/9a879e25f8d44b399e221c8bd78ef976 2023-03-30 03:58:26,935 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/d258a3c37e0d47b29180887f1245d98b 2023-03-30 03:58:27,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148767205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148767206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148767206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,342 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/3273763e40ea4e49a52726214bc2e950 2023-03-30 03:58:27,345 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/9a879e25f8d44b399e221c8bd78ef976 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9a879e25f8d44b399e221c8bd78ef976 2023-03-30 03:58:27,348 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9a879e25f8d44b399e221c8bd78ef976, entries=150, sequenceid=428, filesize=11.9 K 2023-03-30 03:58:27,349 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/d258a3c37e0d47b29180887f1245d98b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/d258a3c37e0d47b29180887f1245d98b 2023-03-30 03:58:27,352 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/d258a3c37e0d47b29180887f1245d98b, entries=150, sequenceid=428, filesize=11.9 K 2023-03-30 03:58:27,352 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/3273763e40ea4e49a52726214bc2e950 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3273763e40ea4e49a52726214bc2e950 2023-03-30 03:58:27,355 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3273763e40ea4e49a52726214bc2e950, entries=150, sequenceid=428, filesize=11.9 K 2023-03-30 03:58:27,356 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e121dc4bd8e8d27540b9c7067395d6e9 in 834ms, sequenceid=428, compaction requested=true 2023-03-30 03:58:27,356 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:27,356 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:27,356 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:27,356 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:27,356 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,356 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:27,356 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:27,357 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:27,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:27,357 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:27,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:27,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@238a9dc9[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:27,359 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:27,359 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:27,359 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:27,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:27,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:27,359 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:27,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:27,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,360 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:27,360 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:27,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:27,360 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:27,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:27,362 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:27,362 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:27,362 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:27,362 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:27,362 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:27,363 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,363 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,363 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:27,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:27,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:27,364 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:27,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:27,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,366 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:27,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:27,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:27,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:27,366 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:27,366 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:27,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:27,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:27,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:27,466 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:27,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:27,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:27,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:27,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:27,469 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:27,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:27,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:27,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:27,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,469 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:27,470 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:27,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:27,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:27,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:27,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,470 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:27,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:27,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:27,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:27,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:27,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:27,471 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:27,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:27,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:27,472 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:27,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:27,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:27,472 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-30 03:58:27,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:27,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:27,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:27,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:27,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:27,472 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:27,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:27,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:27,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148707730, max timestamp=1680148707738Num uniques 859; 2023-03-30 03:58:27,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:27,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148707730, max timestamp=1680148707738Num uniques 859; 2023-03-30 03:58:27,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148707730, max timestamp=1680148707738Num uniques 859; 2023-03-30 03:58:27,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:27,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:27,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148767778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148767779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148767780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,878 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/0a7c65855cb34606b92cbad56dbe4e24 2023-03-30 03:58:27,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148767884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148767887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:27,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:27,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148767887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148768089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148768093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148768095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,286 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/5a37c95f02924910938c1184d5c0497e 2023-03-30 03:58:28,308 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/f9c3c38a31044c139c2783241440ab6b 2023-03-30 03:58:28,313 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/0a7c65855cb34606b92cbad56dbe4e24 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/0a7c65855cb34606b92cbad56dbe4e24 2023-03-30 03:58:28,317 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/0a7c65855cb34606b92cbad56dbe4e24, entries=150, sequenceid=437, filesize=11.9 K 2023-03-30 03:58:28,318 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/5a37c95f02924910938c1184d5c0497e as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5a37c95f02924910938c1184d5c0497e 2023-03-30 03:58:28,322 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5a37c95f02924910938c1184d5c0497e, entries=150, sequenceid=437, filesize=11.9 K 2023-03-30 03:58:28,323 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/f9c3c38a31044c139c2783241440ab6b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f9c3c38a31044c139c2783241440ab6b 2023-03-30 03:58:28,326 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f9c3c38a31044c139c2783241440ab6b, entries=150, sequenceid=437, filesize=11.9 K 2023-03-30 03:58:28,326 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for e121dc4bd8e8d27540b9c7067395d6e9 in 854ms, sequenceid=437, compaction requested=true 2023-03-30 03:58:28,326 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:28,326 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:28,327 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:28,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,328 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,329 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:28,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5052ad8c[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:28,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:28,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,330 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:28,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:28,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:28,330 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,330 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:28,332 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:28,332 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,332 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,332 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:28,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:28,334 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:28,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:28,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:28,334 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:28,334 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:28,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,335 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,335 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,336 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,339 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:28,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,339 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,339 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:28,339 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:28,401 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-03-30 03:58:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:58:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:58:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:28,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 03:58:28,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148768406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148768414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148768414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,439 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:28,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:28,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:28,442 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:28,442 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,442 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:28,442 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:28,443 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,443 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,443 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,443 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,444 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,444 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:28,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:28,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:28,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:28,446 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:28,446 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:28,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:28,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:28,447 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,447 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:28,447 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:28,447 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:28,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:28,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:28,447 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:28,448 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,449 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,449 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:28,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba2b965[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:28,450 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:28,450 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,451 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,451 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:28,451 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:28,451 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:28,451 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,451 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:28,452 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:28,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:28,454 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:28,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:28,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:28,454 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:28,454 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:28,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,455 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,456 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,457 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,458 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:28,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:28,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:28,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,459 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:28,458 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,459 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,459 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:28,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148768515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148768518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148768522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,559 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:28,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:28,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:28,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:28,562 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:28,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:28,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:28,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,562 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,563 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,563 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,563 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,563 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:28,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:28,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:28,564 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,564 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:28,564 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:28,564 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,565 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:28,565 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:28,565 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,565 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,566 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,566 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:28,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@216bc412[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:28,567 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:28,567 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,567 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:28,568 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,568 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:28,568 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:28,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:28,570 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:28,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:28,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:28,570 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:28,570 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:28,571 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,571 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,571 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,572 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,573 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,574 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,574 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,574 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,575 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:28,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,575 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:28,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:28,575 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,575 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,575 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,575 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:28,575 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:28,575 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,575 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,675 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:28,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:28,678 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:28,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:28,678 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:28,679 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,679 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,679 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,679 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,679 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,680 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:28,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:28,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:28,680 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,681 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:28,681 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:28,681 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:28,681 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,682 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:28,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56786123[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:28,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:28,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,683 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,683 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:28,683 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:28,684 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:28,684 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:28,685 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:28,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,685 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:28,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:28,687 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:28,687 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:28,687 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:28,687 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:28,687 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:28,688 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,688 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,688 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,688 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,689 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,691 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:28,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:28,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:28,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,691 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,691 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,691 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148768720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148768725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:28,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148768726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,791 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:28,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:28,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:28,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:28,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:28,795 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:28,795 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,795 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:28,795 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:28,795 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,796 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,796 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:28,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:28,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:28,797 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:28,797 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,797 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:28,797 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:28,797 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:28,797 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,797 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:28,797 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:28,797 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:28,798 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,799 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,799 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:28,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@138388ad[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:28,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:28,803 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,804 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:28,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:28,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:28,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:28,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:28,805 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,805 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,806 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:28,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:28,807 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:28,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:28,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:28,807 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:28,807 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:28,808 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,808 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,808 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,809 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/f11c20c1b8ec45f3bc1b40a05c23edee 2023-03-30 03:58:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,810 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,812 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,812 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:28,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:28,813 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:28,813 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:28,913 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:28,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:28,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:28,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:28,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:28,916 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:28,916 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,916 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:28,916 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:28,916 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,916 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,917 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,917 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,917 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,917 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:28,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:28,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:28,918 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:28,918 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:28,918 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:28,918 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:28,918 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:28,918 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,918 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:28,918 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:28,919 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:28,919 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:28,919 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:28,919 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,919 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:28,919 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:28,919 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,919 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,920 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,920 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:28,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da47229[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:28,921 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:28,921 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,921 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,921 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:28,922 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:28,922 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,922 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:28,922 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:28,922 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,922 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:28,922 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,922 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:28,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:28,924 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:28,924 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:28,924 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:28,924 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:28,924 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:28,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:28,925 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:28,925 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,926 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:28,928 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:28,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:28,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:28,928 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:28,928 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:28,928 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:28,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:28,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:29,028 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:29,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:29,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:29,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:29,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:29,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148769027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:29,031 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:29,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:29,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:29,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:29,031 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,032 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:29,032 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,032 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:29,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:29,032 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,032 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:29,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:29,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:29,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:29,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:29,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:29,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:29,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:29,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:29,033 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:29,033 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:29,033 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. started... 2023-03-30 03:58:29,034 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. as already flushing 2023-03-30 03:58:29,034 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:29,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:29,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:29,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:29,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:29,034 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:29,034 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:29,035 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:29,035 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:29,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:29,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@221066db[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:29,036 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:29,036 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:29,036 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:29,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:29,036 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,036 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:29,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:29,037 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148769031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:29,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:29,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:29,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:29,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148769031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:29,037 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:29,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:29,039 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:29,039 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:29,039 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:29,039 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:29,039 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:29,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:29,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,040 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:29,040 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:29,041 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:29,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:29,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:29,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:29,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:29,043 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:29,043 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:29,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:29,043 DEBUG [Thread-3073] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:29,043 DEBUG [Thread-3073] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:29,068 DEBUG [Thread-3076] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fc0e3f to 127.0.0.1:58033 2023-03-30 03:58:29,068 DEBUG [Thread-3076] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:29,070 DEBUG [Thread-3082] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4486b3c7 to 127.0.0.1:58033 2023-03-30 03:58:29,070 DEBUG [Thread-3082] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:29,071 DEBUG [Thread-3078] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79db57d4 to 127.0.0.1:58033 2023-03-30 03:58:29,071 DEBUG [Thread-3078] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:29,076 DEBUG [Thread-3080] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e56ea94 to 127.0.0.1:58033 2023-03-30 03:58:29,076 DEBUG [Thread-3080] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:29,078 DEBUG [Thread-3074] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f4b7807 to 127.0.0.1:58033 2023-03-30 03:58:29,078 DEBUG [Thread-3074] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:29,143 DEBUG [Thread-3073] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:29,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:29,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/4398c6c348d748dbbbfb58f1c1f13996 2023-03-30 03:58:29,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:29,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39844 deadline: 1680148769537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:29,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39894 deadline: 1680148769541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39910 deadline: 1680148769542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:29,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/1e3017ccee004326ad27c8e167918cd1 2023-03-30 03:58:29,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/f11c20c1b8ec45f3bc1b40a05c23edee as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f11c20c1b8ec45f3bc1b40a05c23edee 2023-03-30 03:58:29,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f11c20c1b8ec45f3bc1b40a05c23edee, entries=200, sequenceid=475, filesize=14.3 K 2023-03-30 03:58:29,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/4398c6c348d748dbbbfb58f1c1f13996 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4398c6c348d748dbbbfb58f1c1f13996 2023-03-30 03:58:29,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4398c6c348d748dbbbfb58f1c1f13996, entries=150, sequenceid=475, filesize=11.9 K 2023-03-30 03:58:29,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/1e3017ccee004326ad27c8e167918cd1 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1e3017ccee004326ad27c8e167918cd1 2023-03-30 03:58:29,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1e3017ccee004326ad27c8e167918cd1, entries=150, sequenceid=475, filesize=11.9 K 2023-03-30 03:58:29,665 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for e121dc4bd8e8d27540b9c7067395d6e9 in 1264ms, sequenceid=475, compaction requested=true 2023-03-30 03:58:29,665 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:29,665 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:29,666 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:29,666 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:58:29,666 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:29,666 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:58:29,667 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:58:29,668 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/A is initiating minor compaction (all files) 2023-03-30 03:58:29,668 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:58:29,668 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/A in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:29,668 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/B is initiating minor compaction (all files) 2023-03-30 03:58:29,668 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1881f37e8a284de9af5894621776e312, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9ffda31578e345c995adc0c5b3644f26, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9a879e25f8d44b399e221c8bd78ef976, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/0a7c65855cb34606b92cbad56dbe4e24, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f11c20c1b8ec45f3bc1b40a05c23edee] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=62.6 K 2023-03-30 03:58:29,668 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/B in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:29,668 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4c9275a7e9dd4601af4397cb555cec9a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e18914a944a244deb18a9b41a0865bba, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/d258a3c37e0d47b29180887f1245d98b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5a37c95f02924910938c1184d5c0497e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4398c6c348d748dbbbfb58f1c1f13996] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=60.2 K 2023-03-30 03:58:29,668 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1881f37e8a284de9af5894621776e312, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680148703099 2023-03-30 03:58:29,668 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4c9275a7e9dd4601af4397cb555cec9a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680148703099 2023-03-30 03:58:29,669 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9ffda31578e345c995adc0c5b3644f26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680148704245 2023-03-30 03:58:29,669 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting e18914a944a244deb18a9b41a0865bba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680148704245 2023-03-30 03:58:29,669 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 9a879e25f8d44b399e221c8bd78ef976, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680148705453 2023-03-30 03:58:29,669 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting d258a3c37e0d47b29180887f1245d98b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680148705453 2023-03-30 03:58:29,669 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 0a7c65855cb34606b92cbad56dbe4e24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680148706581 2023-03-30 03:58:29,670 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 5a37c95f02924910938c1184d5c0497e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680148706581 2023-03-30 03:58:29,670 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting f11c20c1b8ec45f3bc1b40a05c23edee, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680148707769 2023-03-30 03:58:29,670 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 4398c6c348d748dbbbfb58f1c1f13996, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680148707777 2023-03-30 03:58:29,680 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#A#compaction#462 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:29,681 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#B#compaction#463 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:29,769 DEBUG [Thread-3065] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cb7a788 to 127.0.0.1:58033 2023-03-30 03:58:29,769 DEBUG [Thread-3065] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:30,094 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/bdd1ac92f308486d8c15d797dfa5939a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/bdd1ac92f308486d8c15d797dfa5939a 2023-03-30 03:58:30,094 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/cc21824caf5542139b2eec9e644cda33 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/cc21824caf5542139b2eec9e644cda33 2023-03-30 03:58:30,098 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/A of e121dc4bd8e8d27540b9c7067395d6e9 into bdd1ac92f308486d8c15d797dfa5939a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:30,098 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/B of e121dc4bd8e8d27540b9c7067395d6e9 into cc21824caf5542139b2eec9e644cda33(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:30,098 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:30,098 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:30,098 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/A, priority=11, startTime=1680148709665; duration=0sec 2023-03-30 03:58:30,098 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/B, priority=11, startTime=1680148709665; duration=0sec 2023-03-30 03:58:30,098 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:30,098 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:30,098 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 03:58:30,099 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 03:58:30,100 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): e121dc4bd8e8d27540b9c7067395d6e9/C is initiating minor compaction (all files) 2023-03-30 03:58:30,100 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e121dc4bd8e8d27540b9c7067395d6e9/C in TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:30,100 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/15e291603bd4477c9005b3057d480952, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5822736b129f446bad8cf9e9a9872730, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3273763e40ea4e49a52726214bc2e950, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f9c3c38a31044c139c2783241440ab6b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1e3017ccee004326ad27c8e167918cd1] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp, totalSize=60.2 K 2023-03-30 03:58:30,100 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 15e291603bd4477c9005b3057d480952, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680148703099 2023-03-30 03:58:30,100 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 5822736b129f446bad8cf9e9a9872730, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1680148704245 2023-03-30 03:58:30,100 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 3273763e40ea4e49a52726214bc2e950, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680148705453 2023-03-30 03:58:30,101 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting f9c3c38a31044c139c2783241440ab6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680148706581 2023-03-30 03:58:30,101 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 1e3017ccee004326ad27c8e167918cd1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680148707777 2023-03-30 03:58:30,109 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): e121dc4bd8e8d27540b9c7067395d6e9#C#compaction#464 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:30,522 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/9a52a60449a24f549a27945908f97ecf as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9a52a60449a24f549a27945908f97ecf 2023-03-30 03:58:30,530 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e121dc4bd8e8d27540b9c7067395d6e9/C of e121dc4bd8e8d27540b9c7067395d6e9 into 9a52a60449a24f549a27945908f97ecf(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:30,530 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:30,530 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9., storeName=e121dc4bd8e8d27540b9c7067395d6e9/C, priority=11, startTime=1680148709666; duration=0sec 2023-03-30 03:58:30,530 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:30,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:30,540 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:30,540 DEBUG [Thread-3071] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5d7fcf to 127.0.0.1:58033 2023-03-30 03:58:30,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:30,541 DEBUG [Thread-3071] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:30,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:30,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:30,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:30,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:30,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:30,552 DEBUG [Thread-3067] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41ccedcb to 127.0.0.1:58033 2023-03-30 03:58:30,552 DEBUG [Thread-3067] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:30,553 DEBUG [Thread-3063] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02e8c6f6 to 127.0.0.1:58033 2023-03-30 03:58:30,553 DEBUG [Thread-3063] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:30,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/409ff94c8e334693a9fde032b8489816 2023-03-30 03:58:31,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/620a2a0982c64fadad29221f58305e92 2023-03-30 03:58:31,786 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/f644427fc065467f90f1379383cf6994 2023-03-30 03:58:31,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/409ff94c8e334693a9fde032b8489816 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/409ff94c8e334693a9fde032b8489816 2023-03-30 03:58:31,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/409ff94c8e334693a9fde032b8489816, entries=150, sequenceid=488, filesize=11.9 K 2023-03-30 03:58:31,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/620a2a0982c64fadad29221f58305e92 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/620a2a0982c64fadad29221f58305e92 2023-03-30 03:58:31,804 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/620a2a0982c64fadad29221f58305e92, entries=150, sequenceid=488, filesize=11.9 K 2023-03-30 03:58:31,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/f644427fc065467f90f1379383cf6994 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f644427fc065467f90f1379383cf6994 2023-03-30 03:58:31,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f644427fc065467f90f1379383cf6994, entries=150, sequenceid=488, filesize=11.9 K 2023-03-30 03:58:31,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=13.42 KB/13740 for e121dc4bd8e8d27540b9c7067395d6e9 in 1268ms, sequenceid=488, compaction requested=false 2023-03-30 03:58:31,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:33,803 DEBUG [Thread-3069] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e6f624 to 127.0.0.1:58033 2023-03-30 03:58:33,803 DEBUG [Thread-3069] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:33,803 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 03:58:33,804 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-03-30 03:58:33,804 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-03-30 03:58:33,804 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 115 2023-03-30 03:58:33,804 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-03-30 03:58:33,804 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-03-30 03:58:33,804 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 03:58:33,804 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 03:58:33,805 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 2119 2023-03-30 03:58:33,805 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 6353 rows 2023-03-30 03:58:33,805 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 2111 2023-03-30 03:58:33,805 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 6330 rows 2023-03-30 03:58:33,805 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 2128 2023-03-30 03:58:33,805 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 6384 rows 2023-03-30 03:58:33,805 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 2112 2023-03-30 03:58:33,805 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 6335 rows 2023-03-30 03:58:33,806 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(402): scanned 2122 2023-03-30 03:58:33,806 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(403): verified 6364 rows 2023-03-30 03:58:33,806 INFO [Listener at localhost.localdomain/37215] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 03:58:33,806 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a885842 to 127.0.0.1:58033 2023-03-30 03:58:33,806 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:33,809 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 03:58:33,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-30 03:58:33,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:33,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 03:58:33,815 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148713815"}]},"ts":"1680148713815"} 2023-03-30 03:58:33,815 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 03:58:33,835 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 03:58:33,836 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e121dc4bd8e8d27540b9c7067395d6e9, UNASSIGN}] 2023-03-30 03:58:33,837 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e121dc4bd8e8d27540b9c7067395d6e9, UNASSIGN 2023-03-30 03:58:33,837 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=e121dc4bd8e8d27540b9c7067395d6e9, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:33,838 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148713837"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148713837"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148713837"}]},"ts":"1680148713837"} 2023-03-30 03:58:33,839 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:58:33,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 03:58:33,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:33,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing e121dc4bd8e8d27540b9c7067395d6e9, disabling compactions & flushes 2023-03-30 03:58:33,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:33,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:33,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. after waiting 0 ms 2023-03-30 03:58:33,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:33,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing e121dc4bd8e8d27540b9c7067395d6e9 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-30 03:58:33,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=A 2023-03-30 03:58:33,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:33,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=B 2023-03-30 03:58:33,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:33,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e121dc4bd8e8d27540b9c7067395d6e9, store=C 2023-03-30 03:58:33,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:34,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 03:58:34,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/b9d7304534c64f4bb891d8bf176c7615 2023-03-30 03:58:34,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 03:58:34,818 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/c03418a2ba564eacb2cb90c756f2984b 2023-03-30 03:58:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 03:58:35,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/5e2229a76c8d43c4b5f4b6859fbc2577 2023-03-30 03:58:35,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/A/b9d7304534c64f4bb891d8bf176c7615 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/b9d7304534c64f4bb891d8bf176c7615 2023-03-30 03:58:35,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/b9d7304534c64f4bb891d8bf176c7615, entries=100, sequenceid=494, filesize=9.5 K 2023-03-30 03:58:35,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/B/c03418a2ba564eacb2cb90c756f2984b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/c03418a2ba564eacb2cb90c756f2984b 2023-03-30 03:58:35,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/c03418a2ba564eacb2cb90c756f2984b, entries=100, sequenceid=494, filesize=9.5 K 2023-03-30 03:58:35,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/.tmp/C/5e2229a76c8d43c4b5f4b6859fbc2577 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5e2229a76c8d43c4b5f4b6859fbc2577 2023-03-30 03:58:35,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5e2229a76c8d43c4b5f4b6859fbc2577, entries=100, sequenceid=494, filesize=9.5 K 2023-03-30 03:58:35,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for e121dc4bd8e8d27540b9c7067395d6e9 in 1257ms, sequenceid=494, compaction requested=true 2023-03-30 03:58:35,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1a103f11ff07467a954369cf18795111, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/388307fb3bce41629d449946496e8260, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/38ac5dd572f9447f880bc3ff27f9d371, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9d45dbdc97364860b0c8ca0756becc3e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/72225116d88b41d7998a2ff905b4f12c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8d0cd15b3e2342ac9479dd8170497df6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/186654ffefe34a329249224092ac998e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/5cac787d766448498fb08551a8fa617e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e44999edff6441048af54e2fbf5c39af, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8bacd4234dc7449cab20964ebcbbd459, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/2901289125be411faedd9ad278f2deb3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e1fd21216d2f4e62b62837c3809bc87d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/372a7e7bd1f64b278d103ee0709bfd3b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/6913e224afc5484198c96b37015d47ab, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1e2776a69c13404f92de3758afa9cc83, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f3e5f94a414b4875bd4ba7452a2dc4e5, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3801bbc49e284418a4cca0fc8a24e9e6, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/686e0cd2559f468aabf630b32ba177df, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/d87cf86f143b42c0bfffbc3f8e31bb2e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3a489a669db34f24a59c6f02492a55cb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e931ef5e51874a90aa241b21cb7f6d76, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1881f37e8a284de9af5894621776e312, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9ffda31578e345c995adc0c5b3644f26, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9a879e25f8d44b399e221c8bd78ef976, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/0a7c65855cb34606b92cbad56dbe4e24, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f11c20c1b8ec45f3bc1b40a05c23edee] to archive 2023-03-30 03:58:35,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:58:35,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1a103f11ff07467a954369cf18795111 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1a103f11ff07467a954369cf18795111 2023-03-30 03:58:35,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/388307fb3bce41629d449946496e8260 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/388307fb3bce41629d449946496e8260 2023-03-30 03:58:35,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/38ac5dd572f9447f880bc3ff27f9d371 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/38ac5dd572f9447f880bc3ff27f9d371 2023-03-30 03:58:35,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9d45dbdc97364860b0c8ca0756becc3e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9d45dbdc97364860b0c8ca0756becc3e 2023-03-30 03:58:35,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/72225116d88b41d7998a2ff905b4f12c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/72225116d88b41d7998a2ff905b4f12c 2023-03-30 03:58:35,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8d0cd15b3e2342ac9479dd8170497df6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8d0cd15b3e2342ac9479dd8170497df6 2023-03-30 03:58:35,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/186654ffefe34a329249224092ac998e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/186654ffefe34a329249224092ac998e 2023-03-30 03:58:35,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/5cac787d766448498fb08551a8fa617e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/5cac787d766448498fb08551a8fa617e 2023-03-30 03:58:35,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e44999edff6441048af54e2fbf5c39af to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e44999edff6441048af54e2fbf5c39af 2023-03-30 03:58:35,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8bacd4234dc7449cab20964ebcbbd459 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/8bacd4234dc7449cab20964ebcbbd459 2023-03-30 03:58:35,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/2901289125be411faedd9ad278f2deb3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/2901289125be411faedd9ad278f2deb3 2023-03-30 03:58:35,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e1fd21216d2f4e62b62837c3809bc87d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e1fd21216d2f4e62b62837c3809bc87d 2023-03-30 03:58:35,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/372a7e7bd1f64b278d103ee0709bfd3b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/372a7e7bd1f64b278d103ee0709bfd3b 2023-03-30 03:58:35,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/6913e224afc5484198c96b37015d47ab to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/6913e224afc5484198c96b37015d47ab 2023-03-30 03:58:35,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1e2776a69c13404f92de3758afa9cc83 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1e2776a69c13404f92de3758afa9cc83 2023-03-30 03:58:35,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f3e5f94a414b4875bd4ba7452a2dc4e5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f3e5f94a414b4875bd4ba7452a2dc4e5 2023-03-30 03:58:35,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3801bbc49e284418a4cca0fc8a24e9e6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3801bbc49e284418a4cca0fc8a24e9e6 2023-03-30 03:58:35,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/686e0cd2559f468aabf630b32ba177df to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/686e0cd2559f468aabf630b32ba177df 2023-03-30 03:58:35,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/d87cf86f143b42c0bfffbc3f8e31bb2e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/d87cf86f143b42c0bfffbc3f8e31bb2e 2023-03-30 03:58:35,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3a489a669db34f24a59c6f02492a55cb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/3a489a669db34f24a59c6f02492a55cb 2023-03-30 03:58:35,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e931ef5e51874a90aa241b21cb7f6d76 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/e931ef5e51874a90aa241b21cb7f6d76 2023-03-30 03:58:35,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1881f37e8a284de9af5894621776e312 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/1881f37e8a284de9af5894621776e312 2023-03-30 03:58:35,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9ffda31578e345c995adc0c5b3644f26 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9ffda31578e345c995adc0c5b3644f26 2023-03-30 03:58:35,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9a879e25f8d44b399e221c8bd78ef976 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/9a879e25f8d44b399e221c8bd78ef976 2023-03-30 03:58:35,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/0a7c65855cb34606b92cbad56dbe4e24 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/0a7c65855cb34606b92cbad56dbe4e24 2023-03-30 03:58:35,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f11c20c1b8ec45f3bc1b40a05c23edee to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/f11c20c1b8ec45f3bc1b40a05c23edee 2023-03-30 03:58:35,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4cef3da2db7a4355a7563d32c1598362, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/7fd0bb40cf1247f28f55145a7d71b245, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/8aa07406a01a43ad94c6a580b8245e53, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a1d1c39545424c8da3f23bebddfc6284, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9f40e9a09d1d4232929494e9dd968d3a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9bfdbd8124874e4e8e71a5840ce826bb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/fa0af5c83a284f67983264e63ae50b1d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5f3d96ede76246759ff12af4df1f946b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/0d519fcb48c14b3aab05ae8f6a6932aa, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/37b9ca2414a54816bc242a81631285e1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/2861f173d02a437fb6c7c9a30e14332c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4f8ebda4f9ad4ec6b9600b6455402a30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a5da19a37cee491998337ec14ac47e4e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e364fe550cc74834a27dc486b0f47011, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a104489199d048039098408140b28e70, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/1d3420a70848483ab1bae124c0fe4cc1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5bb7a880e8b04111ba1c7432f22824b3, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/41d3d44e6d364b9ab32161af9a1fb051, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/82f741d693454446bc23ddf83bcb03be, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/ab6bbaca3e75478881b8dc7f843938ee, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4c9275a7e9dd4601af4397cb555cec9a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/3285844dfb8c4c429427543e3ea7ea47, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e18914a944a244deb18a9b41a0865bba, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/d258a3c37e0d47b29180887f1245d98b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5a37c95f02924910938c1184d5c0497e, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4398c6c348d748dbbbfb58f1c1f13996] to archive 2023-03-30 03:58:35,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:58:35,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4cef3da2db7a4355a7563d32c1598362 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4cef3da2db7a4355a7563d32c1598362 2023-03-30 03:58:35,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/7fd0bb40cf1247f28f55145a7d71b245 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/7fd0bb40cf1247f28f55145a7d71b245 2023-03-30 03:58:35,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/8aa07406a01a43ad94c6a580b8245e53 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/8aa07406a01a43ad94c6a580b8245e53 2023-03-30 03:58:35,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a1d1c39545424c8da3f23bebddfc6284 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a1d1c39545424c8da3f23bebddfc6284 2023-03-30 03:58:35,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9f40e9a09d1d4232929494e9dd968d3a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9f40e9a09d1d4232929494e9dd968d3a 2023-03-30 03:58:35,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9bfdbd8124874e4e8e71a5840ce826bb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/9bfdbd8124874e4e8e71a5840ce826bb 2023-03-30 03:58:35,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/fa0af5c83a284f67983264e63ae50b1d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/fa0af5c83a284f67983264e63ae50b1d 2023-03-30 03:58:35,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5f3d96ede76246759ff12af4df1f946b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5f3d96ede76246759ff12af4df1f946b 2023-03-30 03:58:35,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/0d519fcb48c14b3aab05ae8f6a6932aa to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/0d519fcb48c14b3aab05ae8f6a6932aa 2023-03-30 03:58:35,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/37b9ca2414a54816bc242a81631285e1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/37b9ca2414a54816bc242a81631285e1 2023-03-30 03:58:35,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/2861f173d02a437fb6c7c9a30e14332c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/2861f173d02a437fb6c7c9a30e14332c 2023-03-30 03:58:35,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4f8ebda4f9ad4ec6b9600b6455402a30 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4f8ebda4f9ad4ec6b9600b6455402a30 2023-03-30 03:58:35,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a5da19a37cee491998337ec14ac47e4e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a5da19a37cee491998337ec14ac47e4e 2023-03-30 03:58:35,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e364fe550cc74834a27dc486b0f47011 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e364fe550cc74834a27dc486b0f47011 2023-03-30 03:58:35,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a104489199d048039098408140b28e70 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/a104489199d048039098408140b28e70 2023-03-30 03:58:35,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/1d3420a70848483ab1bae124c0fe4cc1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/1d3420a70848483ab1bae124c0fe4cc1 2023-03-30 03:58:35,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5bb7a880e8b04111ba1c7432f22824b3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5bb7a880e8b04111ba1c7432f22824b3 2023-03-30 03:58:35,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/41d3d44e6d364b9ab32161af9a1fb051 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/41d3d44e6d364b9ab32161af9a1fb051 2023-03-30 03:58:35,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/82f741d693454446bc23ddf83bcb03be to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/82f741d693454446bc23ddf83bcb03be 2023-03-30 03:58:35,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/ab6bbaca3e75478881b8dc7f843938ee to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/ab6bbaca3e75478881b8dc7f843938ee 2023-03-30 03:58:35,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4c9275a7e9dd4601af4397cb555cec9a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4c9275a7e9dd4601af4397cb555cec9a 2023-03-30 03:58:35,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/3285844dfb8c4c429427543e3ea7ea47 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/3285844dfb8c4c429427543e3ea7ea47 2023-03-30 03:58:35,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e18914a944a244deb18a9b41a0865bba to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/e18914a944a244deb18a9b41a0865bba 2023-03-30 03:58:35,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/d258a3c37e0d47b29180887f1245d98b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/d258a3c37e0d47b29180887f1245d98b 2023-03-30 03:58:35,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5a37c95f02924910938c1184d5c0497e to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/5a37c95f02924910938c1184d5c0497e 2023-03-30 03:58:35,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4398c6c348d748dbbbfb58f1c1f13996 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/4398c6c348d748dbbbfb58f1c1f13996 2023-03-30 03:58:35,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3e419744717545aaa6bb93f90679f5cc, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9e16dbb467a44f4bbad37b2eb8900f18, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1c97eaf6bc5749ab99fc07cdef30fb25, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d5e27e4fa7984453a7bdc1fa2678e7a0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/977d144c1a1842e7a568812f235f5dbb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/6d047c339cb34f40a9fbe2ddb207eecb, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/e6e6f43ba2e644eea0a22f84740a6f30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/ded112327d644c08ba74100db874ace4, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/fc947bb7563545f28786591fb468655c, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a37a54b4d7541d3958007408b518102, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/0b22225ddeee44fbb8ad3659b5467c52, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/81b82a79554e456088e7fac4cfeafc30, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d02e8b3b5a6d4471bec418c3add54781, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/bde30678aa864d1084554c51b19da734, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b34c846e4a134c8b887da549d4935092, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/857968dcf3d3404288848eb0040260de, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d939f59112924083baeb9855e0d1faf1, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/8a8127d1e6574e06aa980a0f5a0fa5bf, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/31a7580654434a4e9500288eb18c8190, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b2053fa1895c41a188bcedc1402cb81a, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/15e291603bd4477c9005b3057d480952, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a21ac45139a441d8c8c6e0050a9d5a0, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5822736b129f446bad8cf9e9a9872730, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3273763e40ea4e49a52726214bc2e950, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f9c3c38a31044c139c2783241440ab6b, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1e3017ccee004326ad27c8e167918cd1] to archive 2023-03-30 03:58:35,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:58:35,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3e419744717545aaa6bb93f90679f5cc to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3e419744717545aaa6bb93f90679f5cc 2023-03-30 03:58:35,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9e16dbb467a44f4bbad37b2eb8900f18 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9e16dbb467a44f4bbad37b2eb8900f18 2023-03-30 03:58:35,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1c97eaf6bc5749ab99fc07cdef30fb25 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1c97eaf6bc5749ab99fc07cdef30fb25 2023-03-30 03:58:35,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d5e27e4fa7984453a7bdc1fa2678e7a0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d5e27e4fa7984453a7bdc1fa2678e7a0 2023-03-30 03:58:35,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/977d144c1a1842e7a568812f235f5dbb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/977d144c1a1842e7a568812f235f5dbb 2023-03-30 03:58:35,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/6d047c339cb34f40a9fbe2ddb207eecb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/6d047c339cb34f40a9fbe2ddb207eecb 2023-03-30 03:58:35,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/e6e6f43ba2e644eea0a22f84740a6f30 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/e6e6f43ba2e644eea0a22f84740a6f30 2023-03-30 03:58:35,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/ded112327d644c08ba74100db874ace4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/ded112327d644c08ba74100db874ace4 2023-03-30 03:58:35,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/fc947bb7563545f28786591fb468655c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/fc947bb7563545f28786591fb468655c 2023-03-30 03:58:35,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a37a54b4d7541d3958007408b518102 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a37a54b4d7541d3958007408b518102 2023-03-30 03:58:35,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/0b22225ddeee44fbb8ad3659b5467c52 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/0b22225ddeee44fbb8ad3659b5467c52 2023-03-30 03:58:35,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/81b82a79554e456088e7fac4cfeafc30 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/81b82a79554e456088e7fac4cfeafc30 2023-03-30 03:58:35,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d02e8b3b5a6d4471bec418c3add54781 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d02e8b3b5a6d4471bec418c3add54781 2023-03-30 03:58:35,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/bde30678aa864d1084554c51b19da734 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/bde30678aa864d1084554c51b19da734 2023-03-30 03:58:35,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b34c846e4a134c8b887da549d4935092 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b34c846e4a134c8b887da549d4935092 2023-03-30 03:58:35,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/857968dcf3d3404288848eb0040260de to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/857968dcf3d3404288848eb0040260de 2023-03-30 03:58:35,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d939f59112924083baeb9855e0d1faf1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/d939f59112924083baeb9855e0d1faf1 2023-03-30 03:58:35,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/8a8127d1e6574e06aa980a0f5a0fa5bf to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/8a8127d1e6574e06aa980a0f5a0fa5bf 2023-03-30 03:58:35,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/31a7580654434a4e9500288eb18c8190 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/31a7580654434a4e9500288eb18c8190 2023-03-30 03:58:35,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b2053fa1895c41a188bcedc1402cb81a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/b2053fa1895c41a188bcedc1402cb81a 2023-03-30 03:58:35,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/15e291603bd4477c9005b3057d480952 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/15e291603bd4477c9005b3057d480952 2023-03-30 03:58:35,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a21ac45139a441d8c8c6e0050a9d5a0 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/4a21ac45139a441d8c8c6e0050a9d5a0 2023-03-30 03:58:35,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5822736b129f446bad8cf9e9a9872730 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5822736b129f446bad8cf9e9a9872730 2023-03-30 03:58:35,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3273763e40ea4e49a52726214bc2e950 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/3273763e40ea4e49a52726214bc2e950 2023-03-30 03:58:35,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f9c3c38a31044c139c2783241440ab6b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f9c3c38a31044c139c2783241440ab6b 2023-03-30 03:58:35,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1e3017ccee004326ad27c8e167918cd1 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/1e3017ccee004326ad27c8e167918cd1 2023-03-30 03:58:35,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/recovered.edits/497.seqid, newMaxSeqId=497, maxSeqId=1 2023-03-30 03:58:35,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9. 2023-03-30 03:58:35,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for e121dc4bd8e8d27540b9c7067395d6e9: 2023-03-30 03:58:35,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:35,374 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=e121dc4bd8e8d27540b9c7067395d6e9, regionState=CLOSED 2023-03-30 03:58:35,374 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148715374"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148715374"}]},"ts":"1680148715374"} 2023-03-30 03:58:35,376 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-30 03:58:35,376 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure e121dc4bd8e8d27540b9c7067395d6e9, server=jenkins-hbase20.apache.org,41157,1680148568851 in 1.5360 sec 2023-03-30 03:58:35,378 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-30 03:58:35,378 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e121dc4bd8e8d27540b9c7067395d6e9, UNASSIGN in 1.5400 sec 2023-03-30 03:58:35,378 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148715378"}]},"ts":"1680148715378"} 2023-03-30 03:58:35,379 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 03:58:35,380 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 03:58:35,381 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5700 sec 2023-03-30 03:58:35,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 03:58:35,922 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-30 03:58:35,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-30 03:58:35,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:35,926 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:35,927 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:35,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-30 03:58:35,935 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:35,939 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/recovered.edits] 2023-03-30 03:58:35,942 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/409ff94c8e334693a9fde032b8489816 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/409ff94c8e334693a9fde032b8489816 2023-03-30 03:58:35,944 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/b9d7304534c64f4bb891d8bf176c7615 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/b9d7304534c64f4bb891d8bf176c7615 2023-03-30 03:58:35,946 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/bdd1ac92f308486d8c15d797dfa5939a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/A/bdd1ac92f308486d8c15d797dfa5939a 2023-03-30 03:58:35,949 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/620a2a0982c64fadad29221f58305e92 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/620a2a0982c64fadad29221f58305e92 2023-03-30 03:58:35,951 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/c03418a2ba564eacb2cb90c756f2984b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/c03418a2ba564eacb2cb90c756f2984b 2023-03-30 03:58:35,953 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/cc21824caf5542139b2eec9e644cda33 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/B/cc21824caf5542139b2eec9e644cda33 2023-03-30 03:58:35,955 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5e2229a76c8d43c4b5f4b6859fbc2577 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/5e2229a76c8d43c4b5f4b6859fbc2577 2023-03-30 03:58:35,955 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9a52a60449a24f549a27945908f97ecf to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/9a52a60449a24f549a27945908f97ecf 2023-03-30 03:58:35,956 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f644427fc065467f90f1379383cf6994 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/C/f644427fc065467f90f1379383cf6994 2023-03-30 03:58:35,959 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/recovered.edits/497.seqid to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9/recovered.edits/497.seqid 2023-03-30 03:58:35,959 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/e121dc4bd8e8d27540b9c7067395d6e9 2023-03-30 03:58:35,959 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:58:35,961 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:35,962 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 03:58:35,964 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 03:58:35,964 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:35,964 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 03:58:35,965 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680148715965"}]},"ts":"9223372036854775807"} 2023-03-30 03:58:35,966 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 03:58:35,966 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e121dc4bd8e8d27540b9c7067395d6e9, NAME => 'TestAcidGuarantees,,1680148686804.e121dc4bd8e8d27540b9c7067395d6e9.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 03:58:35,966 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 03:58:35,966 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680148715966"}]},"ts":"9223372036854775807"} 2023-03-30 03:58:35,968 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 03:58:35,969 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:35,971 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 47 msec 2023-03-30 03:58:36,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-30 03:58:36,030 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-30 03:58:36,048 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 229), OpenFileDescriptor=524 (was 556), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=458 (was 528), ProcessCount=166 (was 166), AvailableMemoryMB=11071 (was 11152) 2023-03-30 03:58:36,063 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=524, MaxFileDescriptor=60000, SystemLoadAverage=458, ProcessCount=166, AvailableMemoryMB=11071 2023-03-30 03:58:36,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:58:36,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:58:36,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:36,067 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 03:58:36,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-30 03:58:36,068 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 03:58:36,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 03:58:36,069 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:36,069 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f empty. 2023-03-30 03:58:36,069 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:36,069 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:58:36,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 03:58:36,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 03:58:36,477 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:58:36,480 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8ebabb5078c369476030f11b847f5c8f, NAME => 'TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp 2023-03-30 03:58:36,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 03:58:36,891 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:58:36,891 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8ebabb5078c369476030f11b847f5c8f, disabling compactions & flushes 2023-03-30 03:58:36,892 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:36,892 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:36,892 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. after waiting 0 ms 2023-03-30 03:58:36,892 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:36,892 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:36,892 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:36,896 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 03:58:36,898 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148716897"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148716897"}]},"ts":"1680148716897"} 2023-03-30 03:58:36,900 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 03:58:36,901 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 03:58:36,901 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148716901"}]},"ts":"1680148716901"} 2023-03-30 03:58:36,902 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 03:58:36,905 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, ASSIGN}] 2023-03-30 03:58:36,908 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, ASSIGN 2023-03-30 03:58:36,908 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=false 2023-03-30 03:58:37,060 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=8ebabb5078c369476030f11b847f5c8f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:37,060 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148717060"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148717060"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148717060"}]},"ts":"1680148717060"} 2023-03-30 03:58:37,061 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:58:37,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 03:58:37,220 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:37,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8ebabb5078c369476030f11b847f5c8f, NAME => 'TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:58:37,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:58:37,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,223 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,224 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A 2023-03-30 03:58:37,224 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A 2023-03-30 03:58:37,225 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:37,225 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8ebabb5078c369476030f11b847f5c8f columnFamilyName A 2023-03-30 03:58:37,226 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(310): Store=8ebabb5078c369476030f11b847f5c8f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:37,226 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,228 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B 2023-03-30 03:58:37,228 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B 2023-03-30 03:58:37,228 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:37,228 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8ebabb5078c369476030f11b847f5c8f columnFamilyName B 2023-03-30 03:58:37,229 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(310): Store=8ebabb5078c369476030f11b847f5c8f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:37,229 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,231 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C 2023-03-30 03:58:37,231 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C 2023-03-30 03:58:37,231 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:37,231 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8ebabb5078c369476030f11b847f5c8f columnFamilyName C 2023-03-30 03:58:37,232 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(310): Store=8ebabb5078c369476030f11b847f5c8f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:37,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:37,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:58:37,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:37,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 03:58:37,240 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 8ebabb5078c369476030f11b847f5c8f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71026994, jitterRate=0.05838468670845032}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:58:37,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:37,241 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., pid=56, masterSystemTime=1680148717213 2023-03-30 03:58:37,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:37,242 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:37,243 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=8ebabb5078c369476030f11b847f5c8f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:37,243 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148717243"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148717243"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148717243"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148717243"}]},"ts":"1680148717243"} 2023-03-30 03:58:37,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-30 03:58:37,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 in 183 msec 2023-03-30 03:58:37,248 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-30 03:58:37,248 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, ASSIGN in 341 msec 2023-03-30 03:58:37,249 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 03:58:37,249 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148717249"}]},"ts":"1680148717249"} 2023-03-30 03:58:37,250 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 03:58:37,252 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 03:58:37,254 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1870 sec 2023-03-30 03:58:38,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 03:58:38,178 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-30 03:58:38,179 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x0b200ba7 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:38,187 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a5c51ec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:38,189 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:38,191 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55486, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:38,192 DEBUG [Listener at localhost.localdomain/37215] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 03:58:38,195 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60060, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 03:58:38,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 03:58:38,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 03:58:38,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-30 03:58:38,615 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 03:58:38,618 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 03:58:38,619 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 03:58:38,623 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-30 03:58:38,625 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, REOPEN/MOVE}] 2023-03-30 03:58:38,626 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, REOPEN/MOVE 2023-03-30 03:58:38,626 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8ebabb5078c369476030f11b847f5c8f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:38,626 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148718626"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148718626"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148718626"}]},"ts":"1680148718626"} 2023-03-30 03:58:38,627 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:58:38,778 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:38,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 8ebabb5078c369476030f11b847f5c8f, disabling compactions & flushes 2023-03-30 03:58:38,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:38,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:38,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. after waiting 0 ms 2023-03-30 03:58:38,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:38,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-30 03:58:38,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:38,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:38,787 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 8ebabb5078c369476030f11b847f5c8f to self. 2023-03-30 03:58:38,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:38,790 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8ebabb5078c369476030f11b847f5c8f, regionState=CLOSED 2023-03-30 03:58:38,790 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148718790"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148718790"}]},"ts":"1680148718790"} 2023-03-30 03:58:38,794 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-30 03:58:38,794 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 in 164 msec 2023-03-30 03:58:38,795 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,41157,1680148568851; forceNewPlan=false, retain=true 2023-03-30 03:58:38,946 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8ebabb5078c369476030f11b847f5c8f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:38,946 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148718946"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148718946"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148718946"}]},"ts":"1680148718946"} 2023-03-30 03:58:38,949 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:58:39,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8ebabb5078c369476030f11b847f5c8f, NAME => 'TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.', STARTKEY => '', ENDKEY => ''} 2023-03-30 03:58:39,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 03:58:39,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,112 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,114 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A 2023-03-30 03:58:39,114 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A 2023-03-30 03:58:39,114 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:39,114 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8ebabb5078c369476030f11b847f5c8f columnFamilyName A 2023-03-30 03:58:39,118 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(310): Store=8ebabb5078c369476030f11b847f5c8f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:39,118 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,120 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B 2023-03-30 03:58:39,120 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B 2023-03-30 03:58:39,120 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:39,120 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8ebabb5078c369476030f11b847f5c8f columnFamilyName B 2023-03-30 03:58:39,121 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(310): Store=8ebabb5078c369476030f11b847f5c8f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:39,121 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,123 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C 2023-03-30 03:58:39,123 DEBUG [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C 2023-03-30 03:58:39,123 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 03:58:39,123 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8ebabb5078c369476030f11b847f5c8f columnFamilyName C 2023-03-30 03:58:39,124 INFO [StoreOpener-8ebabb5078c369476030f11b847f5c8f-1] regionserver.HStore(310): Store=8ebabb5078c369476030f11b847f5c8f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 03:58:39,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 03:58:39,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 8ebabb5078c369476030f11b847f5c8f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72627949, jitterRate=0.08224077522754669}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 03:58:39,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:39,132 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., pid=61, masterSystemTime=1680148719103 2023-03-30 03:58:39,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,134 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8ebabb5078c369476030f11b847f5c8f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,134 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148719134"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680148719134"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680148719134"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680148719134"}]},"ts":"1680148719134"} 2023-03-30 03:58:39,137 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-30 03:58:39,138 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 in 186 msec 2023-03-30 03:58:39,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-30 03:58:39,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, REOPEN/MOVE in 512 msec 2023-03-30 03:58:39,141 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-30 03:58:39,141 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 517 msec 2023-03-30 03:58:39,143 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 944 msec 2023-03-30 03:58:39,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-30 03:58:39,144 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x50131d14 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,149 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5be55982, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,150 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x6722b207 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,156 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f11e84a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,157 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x48502121 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,165 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a1010a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,165 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x08d3d8bc to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,173 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4143b9ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,173 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x7e6ad437 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,181 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@686af57e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,182 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x078cd331 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,189 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@387217bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,190 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x1d74fc48 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,198 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c00a18c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,198 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x070ab5f5 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,207 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3da68fe0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,207 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x2daeb795 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,213 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fd90ac8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,213 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(139): Connect 0x28ac2b09 to 127.0.0.1:58033 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 03:58:39,218 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e63396, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 03:58:39,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:39,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:39,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:39,230 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:39,230 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,230 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:39,230 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:39,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,231 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,232 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,232 DEBUG [hconnection-0x12be1a0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,232 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:39,232 DEBUG [hconnection-0x40165d6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:39,233 DEBUG [hconnection-0x7046ae2f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,233 DEBUG [hconnection-0x334ae8e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:39,233 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:39,234 DEBUG [hconnection-0x218146ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:39,234 DEBUG [hconnection-0x6b23489e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:39,234 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:39,236 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,237 DEBUG [hconnection-0x1a2ed244-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,237 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55506, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,237 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55492, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,237 DEBUG [hconnection-0xf2d3a19-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,238 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55542, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,238 DEBUG [hconnection-0x46b61648-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,237 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55516, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,237 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:39,237 DEBUG [hconnection-0x3df56401-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 03:58:39,240 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:39,238 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,243 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,243 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:39,243 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:39,243 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,243 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:39,243 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:39,246 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55572, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,252 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55552, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,252 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55568, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,253 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55580, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,253 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55582, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 03:58:39,253 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,253 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,254 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:39,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37747ac3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:39,257 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:39,257 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:39,257 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:39,258 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,258 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,258 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:39,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:39,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:39,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:39,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:39,259 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:39,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:39,259 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:39,262 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:39,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:39,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:39,262 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:39,263 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:39,264 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,264 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,264 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,265 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,273 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:39,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:39,273 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,274 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:39,274 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:39,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680148719268, max timestamp=1680148719272Num uniques 859; 2023-03-30 03:58:39,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:39,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680148719268, max timestamp=1680148719272Num uniques 859; 2023-03-30 03:58:39,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:39,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680148719268, max timestamp=1680148719272Num uniques 859; 2023-03-30 03:58:39,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 03:58:39,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148779285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148779285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148779286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148779287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148779287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,374 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:39,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:39,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:39,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:39,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:39,377 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:39,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:39,377 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:39,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,378 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,379 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,379 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:39,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:39,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:39,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:39,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:39,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:39,380 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:39,380 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,381 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:39,381 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:39,381 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:39,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,382 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,382 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:39,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38bd4a60[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:39,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:39,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,384 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,384 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:39,384 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:39,384 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:39,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,384 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:39,386 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,386 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:39,386 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,386 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:39,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:39,388 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:39,388 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:39,388 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:39,389 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:39,389 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:39,390 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,390 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,390 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,390 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148779389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148779389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,392 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148779391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148779391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148779391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,393 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,393 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,393 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,393 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,393 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:39,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:39,394 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,393 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,394 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:39,394 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,394 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:39,394 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:39,394 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,494 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:39,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:39,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:39,496 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:39,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:39,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:39,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,497 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,497 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,498 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:39,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:39,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:39,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:39,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:39,499 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,499 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:39,499 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:39,499 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:39,499 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:39,500 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,500 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,500 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,500 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:39,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46f65264[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:39,501 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:39,501 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,502 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,502 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:39,502 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:39,502 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:39,502 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,502 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:39,503 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,503 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,503 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,503 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:39,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:39,505 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:39,505 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:39,505 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:39,505 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:39,505 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:39,506 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,506 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,506 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,506 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,508 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,509 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,509 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,509 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,509 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:39,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:39,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:39,509 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,510 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:39,510 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:39,509 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,510 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,510 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,510 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,510 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148779592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148779592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148779593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148779593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148779594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,610 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:39,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:39,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:39,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:39,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:39,614 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:39,614 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,614 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:39,614 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:39,615 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,615 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,615 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,615 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,616 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:39,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:39,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:39,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:39,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:39,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,617 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:39,630 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:39,630 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,630 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:39,630 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:39,630 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:39,630 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,630 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:39,630 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:39,630 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,630 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:39,630 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:39,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,631 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,632 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,632 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:39,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f1d2df[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:39,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:39,633 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,633 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:39,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:39,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:39,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,634 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:39,634 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,635 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:39,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:39,637 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:39,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:39,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:39,637 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:39,637 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:39,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,638 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,644 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:39,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,644 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:39,644 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:39,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:39,673 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303308563ee47e7474122b540a34d3acfaa37 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308563ee47e7474122b540a34d3acfaa37 2023-03-30 03:58:39,684 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/8831dfcb3e634cc4888cf0e470d25084 2023-03-30 03:58:39,708 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/9392ca1a2fd7420eafcef68e4a62900d 2023-03-30 03:58:39,744 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:39,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:39,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:39,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:39,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:39,747 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:39,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:39,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:39,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,748 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,748 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:39,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:39,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:39,749 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:39,749 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,749 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:39,749 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:39,749 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:39,749 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,749 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:39,750 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:39,750 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:39,750 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:39,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73470e80[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:39,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:39,752 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,753 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:39,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:39,753 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,753 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:39,753 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:39,754 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,754 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:39,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:39,756 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:39,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:39,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:39,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:39,756 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:39,757 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,757 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,757 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,757 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,758 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:39,761 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,760 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,760 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,761 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:39,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:39,761 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:39,761 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:39,761 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,761 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,761 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,861 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:39,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:39,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:39,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:39,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:39,864 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:39,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:39,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:39,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,864 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,865 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,865 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,865 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,865 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:39,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:39,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:39,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:39,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:39,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:39,866 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:39,866 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:39,867 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:39,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21fbb4c2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:39,868 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:39,868 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,869 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:39,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:39,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:39,870 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,870 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:39,871 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:39,871 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:39,871 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:39,871 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:39,871 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:39,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,872 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,874 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,876 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:39,876 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:39,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:39,876 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,876 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,876 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,876 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,876 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:39,876 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,876 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:39,877 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,877 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,877 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148779895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148779895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148779895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148779896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:39,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148779902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,977 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:39,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:39,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:39,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:39,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:39,979 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:39,979 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,979 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:39,979 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:39,979 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,980 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,989 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,989 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:39,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:39,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:39,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:39,990 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,990 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:39,990 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:39,990 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:39,990 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:39,991 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,991 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:39,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55eab7eb[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:39,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:39,992 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,993 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:39,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:39,993 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:39,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,993 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:39,994 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,994 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,994 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,994 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:39,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:39,995 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:39,995 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:39,995 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:39,995 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:39,995 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:39,996 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,996 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,996 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:39,996 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:39,997 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,998 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:39,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:39,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:39,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:39,999 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:39,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:39,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:39,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:39,999 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:39,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:39,999 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:39,999 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:39,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:39,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:40,099 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:40,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:40,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:40,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:40,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:40,101 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:40,101 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:40,101 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:40,101 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:40,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:40,105 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:40,105 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:40,106 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,107 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:40,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:40,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:40,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:40,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:40,108 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:40,108 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:40,108 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:40,108 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:40,108 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:40,108 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:40,108 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:40,108 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:40,109 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:40,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:40,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:40,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:40,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:40,109 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:40,109 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:40,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:40,110 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:40,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:40,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ce4e0a3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:40,111 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:40,111 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:40,111 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:40,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:40,112 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:40,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,112 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:40,112 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:40,112 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:40,112 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:40,113 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:40,113 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:40,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:40,114 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:40,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:40,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:40,114 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:40,114 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:40,115 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,115 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,115 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:40,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:40,115 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:40,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:40,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:40,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:40,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:40,116 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:40,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:40,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:40,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:40,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:40,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:40,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:40,119 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:40,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:40,119 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:40,119 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:40,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:40,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:40,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:40,119 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:40,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:40,120 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/92d3d7532174495f975c339f54836e26 2023-03-30 03:58:40,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/8831dfcb3e634cc4888cf0e470d25084 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/8831dfcb3e634cc4888cf0e470d25084 2023-03-30 03:58:40,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/8831dfcb3e634cc4888cf0e470d25084, entries=150, sequenceid=16, filesize=25.1 K 2023-03-30 03:58:40,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/9392ca1a2fd7420eafcef68e4a62900d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9392ca1a2fd7420eafcef68e4a62900d 2023-03-30 03:58:40,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9392ca1a2fd7420eafcef68e4a62900d, entries=150, sequenceid=16, filesize=11.6 K 2023-03-30 03:58:40,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/92d3d7532174495f975c339f54836e26 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/92d3d7532174495f975c339f54836e26 2023-03-30 03:58:40,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/92d3d7532174495f975c339f54836e26, entries=150, sequenceid=16, filesize=11.6 K 2023-03-30 03:58:40,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 8ebabb5078c369476030f11b847f5c8f in 887ms, sequenceid=16, compaction requested=false 2023-03-30 03:58:40,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:40,219 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:40,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:40,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:40,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:40,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:40,222 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:40,222 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:40,222 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:40,222 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:40,222 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,222 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:40,223 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:40,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:40,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,223 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:40,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:40,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:40,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:40,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:40,224 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:40,224 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:40,224 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:40,224 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:40,224 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:40,224 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:40,224 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:40,224 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-30 03:58:40,224 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:40,224 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:40,224 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:40,225 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:40,225 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:40,225 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:40,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:40,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:40,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148780405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148780406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148780406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148780408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148780408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148780509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148780509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148780511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148780511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,633 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303308f9d381296684fc3a22d02b162aa1729 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308f9d381296684fc3a22d02b162aa1729 2023-03-30 03:58:40,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148780711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148780711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148780713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:40,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148780714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:40,890 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 03:58:41,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148781012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148781015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148781015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148781015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,037 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/28adc2de37df4a4e90b264d8ff1b4a52 2023-03-30 03:58:41,048 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/692cc93ea70141919b217a7bda062c79 2023-03-30 03:58:41,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148781414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,459 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/692b9450864040748d56a28589c56584 2023-03-30 03:58:41,462 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/28adc2de37df4a4e90b264d8ff1b4a52 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/28adc2de37df4a4e90b264d8ff1b4a52 2023-03-30 03:58:41,467 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/28adc2de37df4a4e90b264d8ff1b4a52, entries=150, sequenceid=52, filesize=25.1 K 2023-03-30 03:58:41,467 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/692cc93ea70141919b217a7bda062c79 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/692cc93ea70141919b217a7bda062c79 2023-03-30 03:58:41,470 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/692cc93ea70141919b217a7bda062c79, entries=150, sequenceid=52, filesize=11.6 K 2023-03-30 03:58:41,471 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/692b9450864040748d56a28589c56584 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/692b9450864040748d56a28589c56584 2023-03-30 03:58:41,474 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/692b9450864040748d56a28589c56584, entries=150, sequenceid=52, filesize=11.6 K 2023-03-30 03:58:41,474 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 8ebabb5078c369476030f11b847f5c8f in 1250ms, sequenceid=52, compaction requested=false 2023-03-30 03:58:41,475 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:41,475 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:41,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:41,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:41,475 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:41,476 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,476 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:41,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@771ccab1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:41,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:41,477 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,478 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:41,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:41,478 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:41,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,478 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:41,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:41,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,479 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:41,479 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:41,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:41,481 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:41,482 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:41,482 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:41,482 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:41,482 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:41,483 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,483 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,483 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,483 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,485 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,486 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:41,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:41,486 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,487 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:41,487 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:41,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,487 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:41,518 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 03:58:41,518 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:41,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:41,518 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:41,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:41,518 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:41,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:41,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148781537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680148721529, max timestamp=1680148721539Num uniques 859; 2023-03-30 03:58:41,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680148721529, max timestamp=1680148721539Num uniques 859; 2023-03-30 03:58:41,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:41,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 03:58:41,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680148721529, max timestamp=1680148721539Num uniques 859; 2023-03-30 03:58:41,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-30 03:58:41,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148781559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148781561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148781561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,587 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:41,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:41,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:41,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:41,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:41,589 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:41,589 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:41,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:41,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:41,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,590 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,591 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,591 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:41,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:41,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:41,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:41,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:41,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:41,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:41,592 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:41,592 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,592 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:41,593 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:41,593 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,593 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:41,593 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:41,593 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,593 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:41,593 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:41,594 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,594 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,595 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:41,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b3d3d34[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:41,596 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:41,596 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,596 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:41,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:41,596 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:41,597 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,597 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:41,597 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,597 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:41,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:41,599 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:41,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:41,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:41,599 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:41,599 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:41,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,600 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,602 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,603 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,604 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,605 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:41,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:41,605 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,605 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:41,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:41,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,605 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,605 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:41,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148781641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148781662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148781664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148781666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,705 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:41,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:41,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:41,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:41,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:41,708 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:41,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:41,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:41,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:41,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,708 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,709 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,709 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:41,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:41,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:41,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:41,710 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,710 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:41,710 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:41,710 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:41,711 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:41,711 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,711 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,712 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,712 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:41,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@be88657[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:41,712 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:41,713 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,713 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:41,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:41,713 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:41,713 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,713 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:41,714 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:41,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,714 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:41,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:41,716 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:41,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:41,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:41,716 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:41,716 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:41,717 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,717 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,717 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,717 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,718 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,719 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,719 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,719 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,719 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:41,719 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,719 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,720 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,720 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,720 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:41,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:41,720 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:41,720 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:41,720 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,820 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:41,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:41,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:41,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:41,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:41,823 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:41,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:41,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:41,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:41,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,823 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,825 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:41,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:41,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:41,826 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,826 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:41,826 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:41,826 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:41,826 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:41,827 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,827 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,828 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,828 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:41,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10928e71[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:41,829 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:41,829 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,829 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:41,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:41,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:41,830 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,830 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:41,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,830 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:41,831 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,831 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:41,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:41,832 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:41,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:41,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:41,833 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:41,833 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:41,833 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,833 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,833 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,834 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,835 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,836 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:41,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:41,836 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,837 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:41,837 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:41,837 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:41,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148781843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148781865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148781866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:41,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148781868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,928 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033057d2c40410fa47fe9fa3e82ef018465f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033057d2c40410fa47fe9fa3e82ef018465f 2023-03-30 03:58:41,937 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:41,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:41,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:41,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:41,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:41,940 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:41,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:41,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:41,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:41,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,940 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,940 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,941 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,941 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:41,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:41,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:41,941 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:41,941 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:41,941 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:41,941 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:41,941 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:41,941 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,942 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:41,942 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:41,942 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:41,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:41,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:41,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:41,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:41,942 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,942 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,943 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,943 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:41,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14a8790[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:41,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:41,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,944 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,944 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:41,944 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:41,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,944 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:41,944 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:41,945 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,945 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,945 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,945 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:41,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:41,947 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:41,947 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:41,947 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:41,947 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:41,947 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:41,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,948 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:41,948 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,949 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:41,951 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:41,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:41,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:41,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:41,951 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:41,951 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:41,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:41,951 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,051 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,054 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,054 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,055 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,055 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,055 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,055 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,056 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,056 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:42,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,056 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,057 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,057 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,057 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,057 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,057 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,057 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,058 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@637cde10[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,059 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,059 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,059 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,059 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,059 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,059 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,060 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,060 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,062 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,062 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,062 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,062 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,062 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,062 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,063 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,063 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,063 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,064 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,065 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,065 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,065 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,066 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,066 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:42,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148782146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,166 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,168 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,168 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,168 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,168 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,169 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:42,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148782167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:42,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148782168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,170 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,170 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,170 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,170 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:42,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:42,171 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148782170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,172 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,173 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,173 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c693616[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,174 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,174 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,175 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,175 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,175 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,176 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,176 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,176 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,176 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,177 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,178 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,178 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,178 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,179 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,180 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,181 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,182 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,181 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:42,182 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,182 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,282 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,284 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,284 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,284 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,284 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,285 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,285 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,286 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:42,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,286 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,286 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,287 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,287 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,287 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,287 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,287 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,288 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,288 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280cc4ec[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,289 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,290 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,290 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,290 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,291 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,292 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,292 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,292 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,293 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,293 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,294 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,294 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,295 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,296 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,296 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,296 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,296 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,297 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,332 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/a789c3c22cab4682854c4520d674db91 2023-03-30 03:58:42,397 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,400 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,400 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,401 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,401 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:42,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,402 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,402 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,402 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,402 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,403 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,403 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,404 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dafb07a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,404 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,404 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,405 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,405 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,405 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,405 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,405 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,405 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,406 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,406 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,407 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,407 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,408 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,408 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,408 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,408 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,408 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,409 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,410 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,411 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,411 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,411 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:42,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,411 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,511 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,514 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,514 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,515 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,515 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:42,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,516 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,516 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,516 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,516 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,516 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,517 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,517 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,518 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78a67d81[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,519 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,519 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,520 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,520 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,520 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,521 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,521 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,523 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,523 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,523 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,524 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,524 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,524 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,525 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,526 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,527 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,527 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,528 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,528 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,528 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,628 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,630 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,630 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,630 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,630 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,631 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,631 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,632 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:42,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,632 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,632 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,632 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,632 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,632 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,632 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,633 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,633 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,633 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,633 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,633 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,634 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d9a2d05[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,635 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,636 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,636 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,636 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,636 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,636 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,637 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,637 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,638 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,638 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,639 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,639 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,639 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,640 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,641 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:42,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,642 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,642 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,643 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:42,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148782651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:42,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148782670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:42,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148782670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:42,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148782676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/7d1de1a913e64233a154ae78d3f9ad92 2023-03-30 03:58:42,743 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,745 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,745 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,745 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,745 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,746 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,746 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,747 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:42,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,748 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,748 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,748 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,748 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,749 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,749 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,749 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,749 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53ffb1fe[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,751 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,751 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,751 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,751 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,751 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,751 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,752 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,752 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,752 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,752 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,754 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,754 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,754 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,754 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,754 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,755 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,755 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,755 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,755 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,756 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,758 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,759 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,759 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,759 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,759 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,759 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,759 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,759 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,759 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,859 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,862 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,863 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,864 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,864 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,864 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,864 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,865 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,865 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5353e79b[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,866 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,866 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,867 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,867 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,867 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,867 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,867 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,868 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,869 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,869 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,869 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,870 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,870 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,870 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,871 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,872 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,873 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,873 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,873 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,873 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,973 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:42,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:42,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:42,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:42,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:42,976 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:42,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:42,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:42,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,976 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,977 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,977 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:42,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:42,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:42,978 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,978 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:42,978 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:42,978 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:42,979 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:42,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1852d205[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:42,980 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:42,980 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,981 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,981 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:42,981 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,981 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:42,981 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:42,981 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:42,982 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,982 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:42,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:42,983 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:42,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:42,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:42,984 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:42,984 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:42,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:42,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,984 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,985 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,986 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:42,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:42,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:42,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:42,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:42,987 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:42,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:42,987 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:42,987 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:42,987 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,087 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:43,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:43,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:43,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:43,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:43,090 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:43,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:43,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:43,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:43,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,091 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:43,091 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:43,091 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:43,091 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,092 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:43,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:43,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:43,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:43,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:43,092 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:43,092 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:43,092 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:43,093 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:43,093 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:43,093 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:43,093 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:43,093 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:43,093 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:43,093 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:43,093 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:43,093 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,093 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:43,093 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:43,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,094 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:43,094 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:43,094 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:43,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:43,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ec260f[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:43,095 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:43,095 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:43,096 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:43,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:43,096 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:43,096 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:43,096 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,096 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:43,096 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:43,097 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,097 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:43,097 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:43,097 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:43,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:43,099 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:43,099 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:43,099 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:43,099 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:43,099 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:43,099 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,099 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,099 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,100 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:43,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:43,101 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,102 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:43,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:43,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:43,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:43,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:43,102 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,102 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:43,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:43,103 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:43,148 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/60d018c81d2c4b0687858c735cb7fb9d 2023-03-30 03:58:43,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/a789c3c22cab4682854c4520d674db91 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/a789c3c22cab4682854c4520d674db91 2023-03-30 03:58:43,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/a789c3c22cab4682854c4520d674db91, entries=200, sequenceid=63, filesize=31.9 K 2023-03-30 03:58:43,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/7d1de1a913e64233a154ae78d3f9ad92 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/7d1de1a913e64233a154ae78d3f9ad92 2023-03-30 03:58:43,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/7d1de1a913e64233a154ae78d3f9ad92, entries=150, sequenceid=63, filesize=11.6 K 2023-03-30 03:58:43,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/60d018c81d2c4b0687858c735cb7fb9d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/60d018c81d2c4b0687858c735cb7fb9d 2023-03-30 03:58:43,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/60d018c81d2c4b0687858c735cb7fb9d, entries=150, sequenceid=63, filesize=11.6 K 2023-03-30 03:58:43,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=136.02 KB/139286 for 8ebabb5078c369476030f11b847f5c8f in 1643ms, sequenceid=63, compaction requested=true 2023-03-30 03:58:43,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:43,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:43,161 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:43,162 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:43,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:43,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:43,162 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:43,162 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1912): 8ebabb5078c369476030f11b847f5c8f/A is initiating minor compaction (all files) 2023-03-30 03:58:43,163 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8ebabb5078c369476030f11b847f5c8f/A in TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:43,163 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/8831dfcb3e634cc4888cf0e470d25084, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/28adc2de37df4a4e90b264d8ff1b4a52, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/a789c3c22cab4682854c4520d674db91] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp, totalSize=82.2 K 2023-03-30 03:58:43,163 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:43,163 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 8831dfcb3e634cc4888cf0e470d25084, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680148719255 2023-03-30 03:58:43,163 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 8ebabb5078c369476030f11b847f5c8f/B is initiating minor compaction (all files) 2023-03-30 03:58:43,163 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8ebabb5078c369476030f11b847f5c8f/B in TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:43,163 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9392ca1a2fd7420eafcef68e4a62900d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/692cc93ea70141919b217a7bda062c79, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/7d1de1a913e64233a154ae78d3f9ad92] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp, totalSize=34.8 K 2023-03-30 03:58:43,163 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting 28adc2de37df4a4e90b264d8ff1b4a52, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680148719285 2023-03-30 03:58:43,163 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 9392ca1a2fd7420eafcef68e4a62900d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680148719255 2023-03-30 03:58:43,163 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] compactions.Compactor(207): Compacting a789c3c22cab4682854c4520d674db91, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680148720402 2023-03-30 03:58:43,164 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 692cc93ea70141919b217a7bda062c79, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680148719285 2023-03-30 03:58:43,164 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 7d1de1a913e64233a154ae78d3f9ad92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680148720407 2023-03-30 03:58:43,168 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 8ebabb5078c369476030f11b847f5c8f#B#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:43,170 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8ebabb5078c369476030f11b847f5c8f#A#compaction#480 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:43,203 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:43,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:43,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:43,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:43,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:43,205 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:43,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:43,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:43,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:43,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:43,206 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,207 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:43,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:43,207 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,207 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:43,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:43,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:43,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:43,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:43,208 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:43,208 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:43,208 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=136.02 KB heapSize=384.46 KB 2023-03-30 03:58:43,208 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:43,208 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:43,208 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:43,208 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:43,208 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:43,208 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:43,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:43,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:43,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148783450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:43,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148783553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,578 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/404ad2404f65429fb8946ade1b88d8de as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/404ad2404f65429fb8946ade1b88d8de 2023-03-30 03:58:43,581 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8ebabb5078c369476030f11b847f5c8f/B of 8ebabb5078c369476030f11b847f5c8f into 404ad2404f65429fb8946ade1b88d8de(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:43,582 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:43,582 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., storeName=8ebabb5078c369476030f11b847f5c8f/B, priority=13, startTime=1680148723162; duration=0sec 2023-03-30 03:58:43,582 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:58:43,582 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 03:58:43,582 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 03:58:43,583 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1912): 8ebabb5078c369476030f11b847f5c8f/C is initiating minor compaction (all files) 2023-03-30 03:58:43,583 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8ebabb5078c369476030f11b847f5c8f/C in TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:43,583 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/92d3d7532174495f975c339f54836e26, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/692b9450864040748d56a28589c56584, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/60d018c81d2c4b0687858c735cb7fb9d] into tmpdir=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp, totalSize=34.8 K 2023-03-30 03:58:43,583 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 92d3d7532174495f975c339f54836e26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680148719255 2023-03-30 03:58:43,583 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 692b9450864040748d56a28589c56584, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680148719285 2023-03-30 03:58:43,583 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] compactions.Compactor(207): Compacting 60d018c81d2c4b0687858c735cb7fb9d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680148720407 2023-03-30 03:58:43,588 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] throttle.PressureAwareThroughputController(145): 8ebabb5078c369476030f11b847f5c8f#C#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 03:58:43,622 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d314415daece41af98d0ebd8252ced18 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d314415daece41af98d0ebd8252ced18 2023-03-30 03:58:43,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:43,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148783662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:43,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148783672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148783676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:43,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148783685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148783756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:43,994 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/2c08692d65804403b02cb1eecc7196c6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/2c08692d65804403b02cb1eecc7196c6 2023-03-30 03:58:43,995 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/8d27ce006eb14c1cb8f3d0c5b859434d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/8d27ce006eb14c1cb8f3d0c5b859434d 2023-03-30 03:58:43,998 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8ebabb5078c369476030f11b847f5c8f/A of 8ebabb5078c369476030f11b847f5c8f into 2c08692d65804403b02cb1eecc7196c6(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:43,998 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8ebabb5078c369476030f11b847f5c8f/C of 8ebabb5078c369476030f11b847f5c8f into 8d27ce006eb14c1cb8f3d0c5b859434d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 03:58:43,998 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:43,998 INFO [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., storeName=8ebabb5078c369476030f11b847f5c8f/C, priority=13, startTime=1680148723162; duration=0sec 2023-03-30 03:58:43,998 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:43,998 INFO [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., storeName=8ebabb5078c369476030f11b847f5c8f/A, priority=13, startTime=1680148723161; duration=0sec 2023-03-30 03:58:43,998 DEBUG [RS:0;jenkins-hbase20:41157-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:43,998 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:58:44,025 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/c980ab7546fc4299b112815d2c63ac49 2023-03-30 03:58:44,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:44,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148784061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,433 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/5c668ec77e074dd9a78ad1e12b53e9eb 2023-03-30 03:58:44,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:44,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148784565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,840 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.10 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/bac69369db3847168e8530ced9a5d3ed 2023-03-30 03:58:44,845 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/c980ab7546fc4299b112815d2c63ac49 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/c980ab7546fc4299b112815d2c63ac49 2023-03-30 03:58:44,847 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/c980ab7546fc4299b112815d2c63ac49, entries=150, sequenceid=99, filesize=25.1 K 2023-03-30 03:58:44,848 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/5c668ec77e074dd9a78ad1e12b53e9eb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/5c668ec77e074dd9a78ad1e12b53e9eb 2023-03-30 03:58:44,850 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/5c668ec77e074dd9a78ad1e12b53e9eb, entries=150, sequenceid=99, filesize=11.6 K 2023-03-30 03:58:44,851 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/bac69369db3847168e8530ced9a5d3ed as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/bac69369db3847168e8530ced9a5d3ed 2023-03-30 03:58:44,854 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/bac69369db3847168e8530ced9a5d3ed, entries=150, sequenceid=99, filesize=11.6 K 2023-03-30 03:58:44,854 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.02 KB/139286, heapSize ~384.41 KB/393640, currentSize=46.96 KB/48090 for 8ebabb5078c369476030f11b847f5c8f in 1646ms, sequenceid=99, compaction requested=false 2023-03-30 03:58:44,854 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:44,854 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:44,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:44,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:44,854 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:44,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:44,855 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:44,855 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:44,856 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:44,856 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:44,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:44,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ab21d2d[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:44,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:44,857 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:44,858 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:44,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:44,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:44,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:44,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:44,858 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,858 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:44,858 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,858 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:44,859 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:44,859 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:44,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:44,861 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:44,861 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:44,861 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:44,861 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:44,861 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:44,862 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,862 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,862 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:44,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:44,862 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:44,863 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:44,865 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,865 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:44,865 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:44,866 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,866 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:44,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:44,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:44,866 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:44,866 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:44,866 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:44,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:44,866 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:44,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:44,866 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,866 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:44,866 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:44,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:44,966 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:44,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:44,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:44,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:44,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:44,969 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:44,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:44,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:44,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:44,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:44,970 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:44,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:44,970 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:44,970 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:44,970 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:44,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:44,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:44,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:44,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:44,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:44,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:44,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:44,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:44,971 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:44,971 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:44,971 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:44,971 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:44,972 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:44,972 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:44,972 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:44,972 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:44,972 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:44,972 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:44,986 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330836921daabdf4415bee7b19464073e06 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330836921daabdf4415bee7b19464073e06 2023-03-30 03:58:44,990 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/699ec9a7438b4f17919cc5a61cc776da 2023-03-30 03:58:45,400 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/056740f697044d8bbcceea0cd32b42c9 2023-03-30 03:58:45,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:45,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:45,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148725596, max timestamp=1680148725613Num uniques 859; 2023-03-30 03:58:45,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148725596, max timestamp=1680148725613Num uniques 859; 2023-03-30 03:58:45,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148725596, max timestamp=1680148725613Num uniques 859; 2023-03-30 03:58:45,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:45,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:45,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:45,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148785658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:45,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148785666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,670 DEBUG [Thread-3653] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:45,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148785681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,685 DEBUG [Thread-3651] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4125 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:45,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:45,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148785688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,692 DEBUG [Thread-3645] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:45,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148785704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,708 DEBUG [Thread-3649] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:45,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:45,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148785761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,807 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/7bc87927311045bd90c9abf78b99184a 2023-03-30 03:58:45,811 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/699ec9a7438b4f17919cc5a61cc776da as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/699ec9a7438b4f17919cc5a61cc776da 2023-03-30 03:58:45,814 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/699ec9a7438b4f17919cc5a61cc776da, entries=150, sequenceid=112, filesize=25.1 K 2023-03-30 03:58:45,815 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/056740f697044d8bbcceea0cd32b42c9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/056740f697044d8bbcceea0cd32b42c9 2023-03-30 03:58:45,818 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/056740f697044d8bbcceea0cd32b42c9, entries=150, sequenceid=112, filesize=11.6 K 2023-03-30 03:58:45,818 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/7bc87927311045bd90c9abf78b99184a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/7bc87927311045bd90c9abf78b99184a 2023-03-30 03:58:45,820 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/7bc87927311045bd90c9abf78b99184a, entries=150, sequenceid=112, filesize=11.6 K 2023-03-30 03:58:45,821 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8ebabb5078c369476030f11b847f5c8f in 850ms, sequenceid=112, compaction requested=true 2023-03-30 03:58:45,821 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:45,821 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:45,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:45,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:45,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:45,821 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:45,822 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:45,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:45,822 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:45,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:45,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@361142df[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:45,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:45,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:45,824 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:45,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:45,824 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:45,824 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:45,824 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:45,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,824 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:45,825 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:45,825 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:45,825 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:45,825 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:45,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:45,827 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:45,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:45,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:45,827 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:45,827 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:45,827 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,827 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,828 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:45,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:45,828 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:45,829 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:45,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:45,830 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:45,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:45,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:45,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:45,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:45,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:45,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:45,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,830 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:45,831 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:45,831 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,830 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:45,831 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:45,931 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:45,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:45,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:45,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:45,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:45,934 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:45,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:45,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:45,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:45,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,934 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:45,935 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:45,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:45,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:45,935 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:45,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:45,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:45,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:45,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:45,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:45,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:45,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:45,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:45,936 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:45,936 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:45,936 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:45,936 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 03:58:45,937 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:45,937 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:45,937 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:45,937 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:45,937 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:45,937 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:45,947 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303302a0e962f199347738aaddfc5ee6b29b3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302a0e962f199347738aaddfc5ee6b29b3 2023-03-30 03:58:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:45,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:45,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:45,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148785982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:46,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:46,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148786085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:46,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:46,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148786287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:46,352 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/6e54e63fe0424375a7eee2cf0cde1f7d 2023-03-30 03:58:46,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:46,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148786589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:46,759 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/9851d16222944c3eafeb7851bd36a204 2023-03-30 03:58:47,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:47,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55582 deadline: 1680148787094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,167 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/89fe83e00f854e34bd816c0ea5795564 2023-03-30 03:58:47,171 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/6e54e63fe0424375a7eee2cf0cde1f7d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/6e54e63fe0424375a7eee2cf0cde1f7d 2023-03-30 03:58:47,173 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/6e54e63fe0424375a7eee2cf0cde1f7d, entries=150, sequenceid=149, filesize=25.3 K 2023-03-30 03:58:47,174 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/9851d16222944c3eafeb7851bd36a204 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9851d16222944c3eafeb7851bd36a204 2023-03-30 03:58:47,176 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9851d16222944c3eafeb7851bd36a204, entries=150, sequenceid=149, filesize=11.8 K 2023-03-30 03:58:47,176 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/89fe83e00f854e34bd816c0ea5795564 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/89fe83e00f854e34bd816c0ea5795564 2023-03-30 03:58:47,183 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/89fe83e00f854e34bd816c0ea5795564, entries=150, sequenceid=149, filesize=11.8 K 2023-03-30 03:58:47,184 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8ebabb5078c369476030f11b847f5c8f in 1248ms, sequenceid=149, compaction requested=true 2023-03-30 03:58:47,184 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:47,184 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:47,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:47,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:47,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:47,184 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:47,185 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,185 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:47,185 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:47,185 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:47,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:47,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385d6ce2[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:47,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:47,186 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:47,187 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:47,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:47,187 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:47,187 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:47,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,187 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:47,187 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:47,188 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,188 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:47,188 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:47,188 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:47,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:47,190 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:47,190 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:47,190 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:47,190 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:47,190 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:47,191 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,191 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:47,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,191 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:47,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:47,192 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:47,193 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:47,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:47,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:47,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:47,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:47,194 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:47,194 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:47,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:47,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:47,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:47,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:47,195 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,195 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:47,195 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:47,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:47,295 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:47,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:47,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:47,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:47,298 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:47,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:47,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:47,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:47,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,298 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:47,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,299 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:47,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:47,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:47,299 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:47,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:47,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:47,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:47,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:47,300 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:47,300 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:47,300 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:47,300 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:47,300 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:47,300 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:47,300 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:47,300 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:47,300 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:47,300 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:47,301 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:47,301 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:47,301 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:47,301 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:47,709 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330b6ee51cf310e4f9aa7f22007f3026311 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b6ee51cf310e4f9aa7f22007f3026311 2023-03-30 03:58:47,713 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=159, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/ff7545dea33042e5acf347286c376fb5 2023-03-30 03:58:48,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:48,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:48,123 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/a4c19ba128774157af80739c8f0b9fc7 2023-03-30 03:58:48,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148728144, max timestamp=1680148728152Num uniques 859; 2023-03-30 03:58:48,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148728144, max timestamp=1680148728152Num uniques 859; 2023-03-30 03:58:48,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148728144, max timestamp=1680148728152Num uniques 859; 2023-03-30 03:58:48,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:48,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:48,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:48,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148788201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:48,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148788305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:48,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148788511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,532 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/418da646af10401586b93dc37b8ff7c6 2023-03-30 03:58:48,535 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/ff7545dea33042e5acf347286c376fb5 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/ff7545dea33042e5acf347286c376fb5 2023-03-30 03:58:48,538 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/ff7545dea33042e5acf347286c376fb5, entries=150, sequenceid=159, filesize=25.3 K 2023-03-30 03:58:48,539 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/a4c19ba128774157af80739c8f0b9fc7 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/a4c19ba128774157af80739c8f0b9fc7 2023-03-30 03:58:48,542 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/a4c19ba128774157af80739c8f0b9fc7, entries=150, sequenceid=159, filesize=11.8 K 2023-03-30 03:58:48,542 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/418da646af10401586b93dc37b8ff7c6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/418da646af10401586b93dc37b8ff7c6 2023-03-30 03:58:48,545 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/418da646af10401586b93dc37b8ff7c6, entries=150, sequenceid=159, filesize=11.8 K 2023-03-30 03:58:48,545 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8ebabb5078c369476030f11b847f5c8f in 1245ms, sequenceid=159, compaction requested=true 2023-03-30 03:58:48,545 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:48,545 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:48,545 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:48,545 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:48,545 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,545 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:48,546 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:48,547 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:48,547 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:48,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:48,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@db4e9b8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:48,548 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:48,548 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:48,548 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:48,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:48,548 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,548 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:48,549 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:48,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:48,551 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:48,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:48,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:48,551 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:48,551 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:48,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,551 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:48,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:48,552 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:48,553 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:48,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:48,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:48,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:48,555 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,555 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:48,555 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:48,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:48,555 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:48,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:48,656 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:48,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:48,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:48,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:48,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:48,658 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:48,658 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:48,658 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:48,658 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:48,658 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,658 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:48,659 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:48,659 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:48,659 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,659 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:48,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:48,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:48,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:48,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:48,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:48,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:48,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:48,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:48,660 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:48,660 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:48,660 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:48,661 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 03:58:48,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:48,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:48,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:48,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:48,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:48,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:48,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:48,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:48,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:48,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148788838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:48,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:48,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148788942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:49,078 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330f536cd3588584f798fc6104b2fdba3e4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f536cd3588584f798fc6104b2fdba3e4 2023-03-30 03:58:49,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:49,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148789145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:49,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:49,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148789450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:49,482 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=196, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/d25001fcb77c4da197fe647a9363c81c 2023-03-30 03:58:49,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:49,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55506 deadline: 1680148789687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:49,689 DEBUG [Thread-3651] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:49,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:49,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55528 deadline: 1680148789704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:49,707 DEBUG [Thread-3645] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:49,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:49,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55516 deadline: 1680148789708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:49,711 DEBUG [Thread-3653] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:49,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:49,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55492 deadline: 1680148789746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:49,752 DEBUG [Thread-3649] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f., hostname=jenkins-hbase20.apache.org,41157,1680148568851, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 03:58:49,890 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/5f93034e5d5746d2aaad3f7358c0a2c8 2023-03-30 03:58:49,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:49,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148789954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,307 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/0d398f1e53794a2781fa52cf81a4549b 2023-03-30 03:58:50,310 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/d25001fcb77c4da197fe647a9363c81c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/d25001fcb77c4da197fe647a9363c81c 2023-03-30 03:58:50,314 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/d25001fcb77c4da197fe647a9363c81c, entries=150, sequenceid=196, filesize=25.3 K 2023-03-30 03:58:50,314 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/5f93034e5d5746d2aaad3f7358c0a2c8 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/5f93034e5d5746d2aaad3f7358c0a2c8 2023-03-30 03:58:50,317 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/5f93034e5d5746d2aaad3f7358c0a2c8, entries=150, sequenceid=196, filesize=11.8 K 2023-03-30 03:58:50,318 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/0d398f1e53794a2781fa52cf81a4549b as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/0d398f1e53794a2781fa52cf81a4549b 2023-03-30 03:58:50,320 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/0d398f1e53794a2781fa52cf81a4549b, entries=150, sequenceid=196, filesize=11.8 K 2023-03-30 03:58:50,321 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8ebabb5078c369476030f11b847f5c8f in 1660ms, sequenceid=196, compaction requested=true 2023-03-30 03:58:50,321 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:50,321 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:50,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:50,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:50,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:50,321 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:50,322 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:50,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:50,322 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:50,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:50,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6705c656[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:50,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:50,324 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:50,324 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:50,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:50,324 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:50,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:50,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:50,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,325 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:50,325 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:50,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:50,325 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:50,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:50,327 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:50,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:50,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:50,327 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:50,327 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:50,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:50,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,328 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:50,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:50,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,329 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:50,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:50,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:50,331 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:50,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:50,331 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:50,331 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:50,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:50,431 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:50,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:50,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:50,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:50,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:50,434 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:50,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:50,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:50,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:50,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,434 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:50,435 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,435 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:50,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:50,435 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:50,435 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:50,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:50,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:50,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:50,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:50,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:50,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:50,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:50,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:50,436 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:50,436 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:50,436 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:50,436 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:50,437 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:50,437 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:50,437 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:50,437 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:50,437 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:50,437 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:50,845 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303301de8f94216a94f8489fce444e305b85c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301de8f94216a94f8489fce444e305b85c 2023-03-30 03:58:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:50,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:51,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148731015, max timestamp=1680148731033Num uniques 859; 2023-03-30 03:58:51,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:51,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148731015, max timestamp=1680148731033Num uniques 859; 2023-03-30 03:58:51,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148731015, max timestamp=1680148731033Num uniques 859; 2023-03-30 03:58:51,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:51,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:51,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:51,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148791108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:51,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:51,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148791212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:51,251 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/99b4030b8cad4fc0bf9e4b85509146c6 2023-03-30 03:58:51,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:51,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148791415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:51,659 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/53fe4a1e256b4650be9c405397156a83 2023-03-30 03:58:51,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148791719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,072 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/382da7500dcd406d9e9b83bd63dc5b61 2023-03-30 03:58:52,075 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/99b4030b8cad4fc0bf9e4b85509146c6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/99b4030b8cad4fc0bf9e4b85509146c6 2023-03-30 03:58:52,078 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/99b4030b8cad4fc0bf9e4b85509146c6, entries=150, sequenceid=206, filesize=25.3 K 2023-03-30 03:58:52,079 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/53fe4a1e256b4650be9c405397156a83 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/53fe4a1e256b4650be9c405397156a83 2023-03-30 03:58:52,081 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/53fe4a1e256b4650be9c405397156a83, entries=150, sequenceid=206, filesize=11.8 K 2023-03-30 03:58:52,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/382da7500dcd406d9e9b83bd63dc5b61 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/382da7500dcd406d9e9b83bd63dc5b61 2023-03-30 03:58:52,084 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/382da7500dcd406d9e9b83bd63dc5b61, entries=150, sequenceid=206, filesize=11.8 K 2023-03-30 03:58:52,085 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8ebabb5078c369476030f11b847f5c8f in 1649ms, sequenceid=206, compaction requested=true 2023-03-30 03:58:52,085 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:52,085 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:52,085 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:52,085 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:52,085 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,085 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:52,085 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:52,086 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:52,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:52,086 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:52,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:52,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b3e67d3[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:52,087 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:52,087 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:52,088 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:52,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:52,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:52,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:52,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:52,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,088 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:52,088 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:52,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:52,088 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:52,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:52,090 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:52,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:52,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:52,090 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:52,090 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:52,091 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,091 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,091 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:52,091 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:52,092 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:52,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:52,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:52,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:52,094 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:52,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:52,094 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,094 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:52,094 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,194 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:52,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:52,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:52,197 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:52,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:52,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:52,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:52,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,197 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:52,198 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:52,198 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:52,198 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:52,198 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:52,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:52,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:52,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:52,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:52,199 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:52,199 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:52,199 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 03:58:52,199 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:52,199 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:52,199 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:52,199 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:52,200 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:52,200 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:52,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:52,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:52,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:52,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148792254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:52,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148792358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:52,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148792560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:52,611 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303303ab7682ea4684d0c8537f9b8797993b9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303ab7682ea4684d0c8537f9b8797993b9 2023-03-30 03:58:52,615 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/3ec193a48fc14688b50fcc4ea8ab2e33 2023-03-30 03:58:52,621 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/c0f510b4a1bf4163b371a09e82491a91 2023-03-30 03:58:52,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:52,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148792864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,029 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/743cb74560034b619da34af69c201b19 2023-03-30 03:58:53,032 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/3ec193a48fc14688b50fcc4ea8ab2e33 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/3ec193a48fc14688b50fcc4ea8ab2e33 2023-03-30 03:58:53,034 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/3ec193a48fc14688b50fcc4ea8ab2e33, entries=150, sequenceid=243, filesize=25.3 K 2023-03-30 03:58:53,035 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/c0f510b4a1bf4163b371a09e82491a91 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/c0f510b4a1bf4163b371a09e82491a91 2023-03-30 03:58:53,037 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/c0f510b4a1bf4163b371a09e82491a91, entries=150, sequenceid=243, filesize=11.8 K 2023-03-30 03:58:53,038 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/743cb74560034b619da34af69c201b19 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/743cb74560034b619da34af69c201b19 2023-03-30 03:58:53,040 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/743cb74560034b619da34af69c201b19, entries=150, sequenceid=243, filesize=11.8 K 2023-03-30 03:58:53,041 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8ebabb5078c369476030f11b847f5c8f in 842ms, sequenceid=243, compaction requested=true 2023-03-30 03:58:53,041 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:53,041 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:53,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:53,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:53,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:53,041 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:53,042 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:53,042 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:53,043 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:53,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:53,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d73a2a[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:53,044 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:53,044 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:53,044 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:53,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:53,044 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:53,044 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:53,045 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:53,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,045 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:53,046 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:53,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:53,046 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:53,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:53,048 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:53,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:53,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:53,048 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:53,048 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:53,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,049 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:53,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:53,049 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:53,050 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:53,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:53,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:53,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:53,052 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:53,051 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:53,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:53,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:53,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:53,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:53,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:53,052 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:53,052 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,052 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:53,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:53,152 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:53,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:53,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:53,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:53,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:53,155 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:53,155 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:53,155 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:53,155 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:53,155 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,155 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:53,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,156 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:53,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:53,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,156 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:53,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:53,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:53,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:53,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:53,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:53,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:53,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:53,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:53,157 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:53,157 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:53,157 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:53,157 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:53,157 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:53,157 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:53,158 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:53,158 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:53,158 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:53,158 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:53,167 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033051173181a905435584af096883297bc2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033051173181a905435584af096883297bc2 2023-03-30 03:58:53,171 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=253, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/c9f4df2c90c047fb83663838d3ab8229 2023-03-30 03:58:53,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:53,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:53,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148733389, max timestamp=1680148733430Num uniques 859; 2023-03-30 03:58:53,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:53,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148733389, max timestamp=1680148733430Num uniques 859; 2023-03-30 03:58:53,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:53,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148733389, max timestamp=1680148733430Num uniques 859; 2023-03-30 03:58:53,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:53,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148793511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,578 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/b6d5c7734f494cedb4b9efd6c39c5371 2023-03-30 03:58:53,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:53,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148793617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:53,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148793821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:53,985 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/5369c341305c41cfb23d79d930aa504d 2023-03-30 03:58:53,989 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/c9f4df2c90c047fb83663838d3ab8229 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/c9f4df2c90c047fb83663838d3ab8229 2023-03-30 03:58:53,991 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/c9f4df2c90c047fb83663838d3ab8229, entries=150, sequenceid=253, filesize=25.3 K 2023-03-30 03:58:53,997 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/b6d5c7734f494cedb4b9efd6c39c5371 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/b6d5c7734f494cedb4b9efd6c39c5371 2023-03-30 03:58:54,000 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/b6d5c7734f494cedb4b9efd6c39c5371, entries=150, sequenceid=253, filesize=11.8 K 2023-03-30 03:58:54,000 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/5369c341305c41cfb23d79d930aa504d as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/5369c341305c41cfb23d79d930aa504d 2023-03-30 03:58:54,003 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/5369c341305c41cfb23d79d930aa504d, entries=150, sequenceid=253, filesize=11.8 K 2023-03-30 03:58:54,003 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8ebabb5078c369476030f11b847f5c8f in 846ms, sequenceid=253, compaction requested=true 2023-03-30 03:58:54,004 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:54,004 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:54,004 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:54,004 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:54,004 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,004 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:54,004 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:54,005 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,005 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:54,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,006 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:54,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:54,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2284eba1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:54,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:54,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,007 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,007 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:54,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,007 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:54,007 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:54,007 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:54,008 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:54,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:54,008 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:54,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:54,010 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:54,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:54,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:54,010 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:54,010 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:54,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,011 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:54,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:54,011 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,012 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:54,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:54,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:54,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:54,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:54,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:54,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,013 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:54,014 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:54,013 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:54,014 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:54,014 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:54,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:54,114 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:54,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:54,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:54,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:54,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:54,117 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:54,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:54,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:54,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:54,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,117 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:54,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,118 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:54,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:54,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,118 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:54,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:54,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:54,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:54,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:54,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:54,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:54,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:54,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:54,119 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:54,119 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:54,119 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:54,119 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 03:58:54,119 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:54,120 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:54,120 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:54,120 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:54,120 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:54,120 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:54,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:54,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:54,132 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303309530c105b6c84b7b805f136798ffc93c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309530c105b6c84b7b805f136798ffc93c 2023-03-30 03:58:54,136 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/ee30d47787a04d639c20f58fc97cd306 2023-03-30 03:58:54,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:54,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148794153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:54,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148794257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:54,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148794460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,543 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/67b9be0d3ca24c0e863f0233c5658141 2023-03-30 03:58:54,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:54,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148794764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,951 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/3a5ba214014741e1bfcd60d17330c515 2023-03-30 03:58:54,955 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/ee30d47787a04d639c20f58fc97cd306 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/ee30d47787a04d639c20f58fc97cd306 2023-03-30 03:58:54,957 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/ee30d47787a04d639c20f58fc97cd306, entries=150, sequenceid=290, filesize=25.4 K 2023-03-30 03:58:54,958 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/67b9be0d3ca24c0e863f0233c5658141 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/67b9be0d3ca24c0e863f0233c5658141 2023-03-30 03:58:54,960 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/67b9be0d3ca24c0e863f0233c5658141, entries=150, sequenceid=290, filesize=11.9 K 2023-03-30 03:58:54,960 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/3a5ba214014741e1bfcd60d17330c515 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/3a5ba214014741e1bfcd60d17330c515 2023-03-30 03:58:54,963 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/3a5ba214014741e1bfcd60d17330c515, entries=150, sequenceid=290, filesize=11.9 K 2023-03-30 03:58:54,963 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8ebabb5078c369476030f11b847f5c8f in 844ms, sequenceid=290, compaction requested=true 2023-03-30 03:58:54,963 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:54,963 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:54,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:54,963 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:54,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:54,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:54,964 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,964 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:54,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,965 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:54,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:54,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7abed1[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:54,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:54,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,966 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:54,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:54,966 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:54,966 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,967 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:54,967 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:54,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:54,969 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:54,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:54,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:54,969 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:54,969 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:54,970 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,970 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:54,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,970 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,970 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:54,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:54,971 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:54,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:54,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:54,973 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:54,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:54,973 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:54,973 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:54,973 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:55,073 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:55,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:55,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:55,076 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:55,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:55,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:55,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:55,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:55,076 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:55,077 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:55,077 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:55,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:55,077 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:55,077 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:55,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:55,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:55,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:55,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:55,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:55,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:55,082 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:55,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:55,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:55,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:55,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:55,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:55,082 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:55,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:55,267 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:55,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148735295, max timestamp=1680148735333Num uniques 859; 2023-03-30 03:58:55,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:55,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148735295, max timestamp=1680148735333Num uniques 859; 2023-03-30 03:58:55,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148735295, max timestamp=1680148735333Num uniques 859; 2023-03-30 03:58:55,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:55,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:55,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:55,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148795403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:55,496 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330ad1f0da5f57549018724aa546bc0f7c6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330ad1f0da5f57549018724aa546bc0f7c6 2023-03-30 03:58:55,499 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=300, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/0fa743e083804dbc8b3ef0a4d4855739 2023-03-30 03:58:55,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:55,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148795507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:55,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148795711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:55,907 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/6b6016bdb888430f990a0e7c52192d87 2023-03-30 03:58:56,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148796015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,315 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/1f987d992b044e2697c427f4373c23bb 2023-03-30 03:58:56,318 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/0fa743e083804dbc8b3ef0a4d4855739 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/0fa743e083804dbc8b3ef0a4d4855739 2023-03-30 03:58:56,320 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/0fa743e083804dbc8b3ef0a4d4855739, entries=150, sequenceid=300, filesize=25.4 K 2023-03-30 03:58:56,321 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/6b6016bdb888430f990a0e7c52192d87 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/6b6016bdb888430f990a0e7c52192d87 2023-03-30 03:58:56,323 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/6b6016bdb888430f990a0e7c52192d87, entries=150, sequenceid=300, filesize=11.9 K 2023-03-30 03:58:56,324 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/1f987d992b044e2697c427f4373c23bb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/1f987d992b044e2697c427f4373c23bb 2023-03-30 03:58:56,326 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/1f987d992b044e2697c427f4373c23bb, entries=150, sequenceid=300, filesize=11.9 K 2023-03-30 03:58:56,327 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8ebabb5078c369476030f11b847f5c8f in 1245ms, sequenceid=300, compaction requested=true 2023-03-30 03:58:56,327 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:56,327 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:56,327 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:56,327 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:56,327 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:56,330 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:56,331 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,331 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:56,331 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:56,331 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:56,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:56,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43524c92[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:56,332 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:56,332 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:56,333 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:56,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:56,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:56,333 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,333 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:56,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:56,333 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:56,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:56,334 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:56,334 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:56,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:56,336 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:56,336 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:56,336 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:56,336 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:56,336 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:56,337 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,337 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,337 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:56,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:56,337 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:56,338 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:56,355 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,355 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:56,355 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:56,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:56,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:56,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:56,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:56,355 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:56,355 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:56,355 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:56,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:56,355 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:56,355 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:56,356 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,356 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:56,356 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:56,356 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,456 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:56,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:56,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:56,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:56,458 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:56,458 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:56,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:56,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:56,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,459 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:56,460 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:56,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:56,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,460 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:56,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:56,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:56,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:56,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:56,461 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:56,461 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:56,461 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:56,461 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:56,461 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:56,461 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:56,461 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:56,461 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 03:58:56,461 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:56,461 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:56,461 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:56,462 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:56,462 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:56,462 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:56,471 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033093b1fc6658294b27be3a229cff239df2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033093b1fc6658294b27be3a229cff239df2 2023-03-30 03:58:56,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:56,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:56,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:56,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148796545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:56,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148796650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148796855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:56,876 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/11ec28b52ac9414bb6beef2b8edf20a6 2023-03-30 03:58:57,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:57,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148797160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,284 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/3d059ea5612a4eb2a467fb0ab0546801 2023-03-30 03:58:57,291 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/8edfd59c927a4ff898fa7d2f4210cd99 2023-03-30 03:58:57,294 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/11ec28b52ac9414bb6beef2b8edf20a6 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/11ec28b52ac9414bb6beef2b8edf20a6 2023-03-30 03:58:57,297 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/11ec28b52ac9414bb6beef2b8edf20a6, entries=150, sequenceid=337, filesize=25.4 K 2023-03-30 03:58:57,297 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/3d059ea5612a4eb2a467fb0ab0546801 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/3d059ea5612a4eb2a467fb0ab0546801 2023-03-30 03:58:57,300 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/3d059ea5612a4eb2a467fb0ab0546801, entries=150, sequenceid=337, filesize=11.9 K 2023-03-30 03:58:57,300 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/8edfd59c927a4ff898fa7d2f4210cd99 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/8edfd59c927a4ff898fa7d2f4210cd99 2023-03-30 03:58:57,302 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/8edfd59c927a4ff898fa7d2f4210cd99, entries=150, sequenceid=337, filesize=11.9 K 2023-03-30 03:58:57,303 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8ebabb5078c369476030f11b847f5c8f in 842ms, sequenceid=337, compaction requested=true 2023-03-30 03:58:57,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:57,303 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:57,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:57,304 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:57,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:57,304 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:57,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:57,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e528b8[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:57,305 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:57,305 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:57,306 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:57,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:57,306 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:57,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,307 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:57,307 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:57,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:57,309 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:57,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:57,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:57,309 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:57,309 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:57,309 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,309 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:57,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,310 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:57,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:57,310 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,311 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,312 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,312 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:57,312 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:57,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:57,312 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:57,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:57,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:57,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:57,312 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:57,312 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:57,312 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:57,312 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:57,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:57,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:57,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:57,313 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,413 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:57,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:57,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:57,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:57,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:57,416 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:57,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:57,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:57,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:57,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,416 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:57,417 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,417 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:57,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:57,417 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,417 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:57,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:57,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:57,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:57,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:57,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:57,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:57,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:57,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:57,418 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:57,418 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:57,418 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:57,419 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 03:58:57,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:57,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:57,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:57,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:57,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:57,419 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:57,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:57,666 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:57,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148737714, max timestamp=1680148737742Num uniques 859; 2023-03-30 03:58:57,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:57,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148737714, max timestamp=1680148737742Num uniques 859; 2023-03-30 03:58:57,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680148737714, max timestamp=1680148737742Num uniques 859; 2023-03-30 03:58:57,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:57,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 03:58:57,830 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303309e76eac841304ec890b2e71e4b9d6c46 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309e76eac841304ec890b2e71e4b9d6c46 2023-03-30 03:58:57,834 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/cac7cb410480471e851a410c3bd8c382 2023-03-30 03:58:57,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:57,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148797834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:57,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:57,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148797940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:58,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148798143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,240 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/41acef4f506442cdaff056ce6c74935c 2023-03-30 03:58:58,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:58,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148798448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,648 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/521548cb9ac54be6830065efc7b0cec3 2023-03-30 03:58:58,651 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/cac7cb410480471e851a410c3bd8c382 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/cac7cb410480471e851a410c3bd8c382 2023-03-30 03:58:58,653 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/cac7cb410480471e851a410c3bd8c382, entries=150, sequenceid=347, filesize=25.4 K 2023-03-30 03:58:58,654 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/41acef4f506442cdaff056ce6c74935c as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/41acef4f506442cdaff056ce6c74935c 2023-03-30 03:58:58,657 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/41acef4f506442cdaff056ce6c74935c, entries=150, sequenceid=347, filesize=11.9 K 2023-03-30 03:58:58,657 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/521548cb9ac54be6830065efc7b0cec3 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/521548cb9ac54be6830065efc7b0cec3 2023-03-30 03:58:58,660 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/521548cb9ac54be6830065efc7b0cec3, entries=150, sequenceid=347, filesize=11.9 K 2023-03-30 03:58:58,661 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8ebabb5078c369476030f11b847f5c8f in 1242ms, sequenceid=347, compaction requested=true 2023-03-30 03:58:58,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:58,661 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:58,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:58,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:58,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:58,661 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:58,662 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:58,662 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:58,662 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:58,662 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:58,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:58,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:58,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b90ca6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:58,663 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:58,663 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:58,664 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:58,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:58,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:58,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:58,664 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:58,664 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,664 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:58,665 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,665 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:58,665 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:58,665 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:58,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:58,667 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:58,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:58,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:58,667 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:58,667 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:58,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:58,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:58,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,668 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:58,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:58,668 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:58,669 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,670 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:58,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:58,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:58,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:58,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:58,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:58,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:58,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:58,671 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,671 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:58,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:58,671 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:58,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:58,671 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:58,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:58,771 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:58,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:58,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-30 03:58:58,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 03:58:58,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 03:58:58,774 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 03:58:58,774 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:58,774 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 03:58:58,774 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 03:58:58,775 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,775 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:58,775 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:58,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:58,775 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:58,775 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:58,776 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 03:58:58,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 03:58:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:58,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 03:58:58,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 03:58:58,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 03:58:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 03:58:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 03:58:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 03:58:58,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 03:58:58,777 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 03:58:58,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:58,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. started... 2023-03-30 03:58:58,777 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 03:58:58,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:58,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:58,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:58,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:58,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:58,777 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 03:58:58,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:58,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. as already flushing 2023-03-30 03:58:59,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:59,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41157] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148798998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:59,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148799105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,190 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303309fd922802b98472d8319f7786dc1fcc7 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309fd922802b98472d8319f7786dc1fcc7 2023-03-30 03:58:59,193 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=384, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/2fe927f1a7024a5aae8ed781247d4ced 2023-03-30 03:58:59,225 DEBUG [Thread-3664] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28ac2b09 to 127.0.0.1:58033 2023-03-30 03:58:59,225 DEBUG [Thread-3664] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:59,228 DEBUG [Thread-3658] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d74fc48 to 127.0.0.1:58033 2023-03-30 03:58:59,228 DEBUG [Thread-3658] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:59,229 DEBUG [Thread-3662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2daeb795 to 127.0.0.1:58033 2023-03-30 03:58:59,229 DEBUG [Thread-3662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:59,229 DEBUG [Thread-3656] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x078cd331 to 127.0.0.1:58033 2023-03-30 03:58:59,229 DEBUG [Thread-3656] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:59,233 DEBUG [Thread-3660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x070ab5f5 to 127.0.0.1:58033 2023-03-30 03:58:59,233 DEBUG [Thread-3660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:59,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:59,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148799309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,601 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/98e7e88713a645568b029378b88de9ec 2023-03-30 03:58:59,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 03:58:59,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55582 deadline: 1680148799612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,614 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/91a3cbd7e03d444db567319685eee7fb 2023-03-30 03:58:59,619 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/2fe927f1a7024a5aae8ed781247d4ced as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/2fe927f1a7024a5aae8ed781247d4ced 2023-03-30 03:58:59,624 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/2fe927f1a7024a5aae8ed781247d4ced, entries=150, sequenceid=384, filesize=25.4 K 2023-03-30 03:58:59,624 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/98e7e88713a645568b029378b88de9ec as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/98e7e88713a645568b029378b88de9ec 2023-03-30 03:58:59,632 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/98e7e88713a645568b029378b88de9ec, entries=150, sequenceid=384, filesize=11.9 K 2023-03-30 03:58:59,633 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/91a3cbd7e03d444db567319685eee7fb as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/91a3cbd7e03d444db567319685eee7fb 2023-03-30 03:58:59,637 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/91a3cbd7e03d444db567319685eee7fb, entries=150, sequenceid=384, filesize=11.9 K 2023-03-30 03:58:59,638 INFO [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8ebabb5078c369476030f11b847f5c8f in 861ms, sequenceid=384, compaction requested=true 2023-03-30 03:58:59,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:58:59,638 DEBUG [rs(jenkins-hbase20.apache.org,41157,1680148568851)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:58:59,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 03:58:59,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 03:58:59,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 03:58:59,638 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 03:58:59,640 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:59,640 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:59,640 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:59,640 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 03:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:59,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41157,1680148568851' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 03:58:59,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@498bb8b6[Count = 0] remaining members to acquire global barrier 2023-03-30 03:58:59,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 03:58:59,642 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:59,643 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:59,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:59,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 03:58:59,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 03:58:59,643 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41157,1680148568851' in zk 2023-03-30 03:58:59,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,643 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 03:58:59,644 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 03:58:59,644 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:59,644 DEBUG [member: 'jenkins-hbase20.apache.org,41157,1680148568851' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 03:58:59,644 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41157,1680148568851': 2023-03-30 03:58:59,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41157,1680148568851' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 03:58:59,648 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 03:58:59,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 03:58:59,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 03:58:59,648 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 03:58:59,649 INFO [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 03:58:59,650 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:59,650 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:59,650 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 03:58:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 03:58:59,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:59,651 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 03:58:59,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 03:58:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:59,652 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 03:58:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 03:58:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:59,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 03:58:59,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:58:59,655 DEBUG [(jenkins-hbase20.apache.org,37051,1680148567795)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 03:58:59,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 03:58:59,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 03:58:59,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 03:58:59,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:59,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 03:58:59,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 03:58:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 03:58:59,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 03:58:59,655 DEBUG [Thread-3655] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 03:58:59,656 DEBUG [Thread-3655] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 03:58:59,655 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 03:58:59,656 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 03:58:59,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 03:58:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 03:58:59,756 DEBUG [Thread-3655] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 03:58:59,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 03:58:59,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41157] regionserver.HRegion(9158): Flush requested on 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:58:59,766 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 03:58:59,767 DEBUG [Thread-3645] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50131d14 to 127.0.0.1:58033 2023-03-30 03:58:59,767 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:58:59,767 DEBUG [Thread-3645] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:59,767 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:59,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:58:59,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:59,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:58:59,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:58:59,772 DEBUG [Thread-3651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08d3d8bc to 127.0.0.1:58033 2023-03-30 03:58:59,772 DEBUG [Thread-3651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:59,779 DEBUG [Thread-3653] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e6ad437 to 127.0.0.1:58033 2023-03-30 03:58:59,779 DEBUG [Thread-3653] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:58:59,848 DEBUG [Thread-3649] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48502121 to 127.0.0.1:58033 2023-03-30 03:58:59,849 DEBUG [Thread-3649] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:59:00,121 DEBUG [Thread-3647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6722b207 to 127.0.0.1:58033 2023-03-30 03:59:00,121 DEBUG [Thread-3647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:59:00,121 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 03:59:00,122 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-03-30 03:59:00,122 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 262 2023-03-30 03:59:00,122 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-03-30 03:59:00,122 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-03-30 03:59:00,122 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-03-30 03:59:00,122 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 03:59:00,123 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4669 2023-03-30 03:59:00,123 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4759 2023-03-30 03:59:00,123 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4582 2023-03-30 03:59:00,123 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4676 2023-03-30 03:59:00,123 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(398): read 4774 2023-03-30 03:59:00,123 INFO [Listener at localhost.localdomain/37215] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 03:59:00,124 INFO [Listener at localhost.localdomain/37215] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 03:59:00,124 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b200ba7 to 127.0.0.1:58033 2023-03-30 03:59:00,124 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:59:00,125 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 03:59:00,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-30 03:59:00,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 03:59:00,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 03:59:00,130 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148740130"}]},"ts":"1680148740130"} 2023-03-30 03:59:00,131 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 03:59:00,132 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 03:59:00,133 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, UNASSIGN}] 2023-03-30 03:59:00,134 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, UNASSIGN 2023-03-30 03:59:00,134 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=8ebabb5078c369476030f11b847f5c8f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:59:00,135 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148740134"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680148740134"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680148740134"}]},"ts":"1680148740134"} 2023-03-30 03:59:00,135 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851}] 2023-03-30 03:59:00,185 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303306188898d37624e219035ace8543f7664 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306188898d37624e219035ace8543f7664 2023-03-30 03:59:00,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 03:59:00,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:59:00,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 8ebabb5078c369476030f11b847f5c8f, disabling compactions & flushes 2023-03-30 03:59:00,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:59:00,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 03:59:00,593 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/df110fe3b06d4e36aae8c7586b49e9a9 2023-03-30 03:59:00,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 03:59:01,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/c0163465c0384d99b3c91fe8ec4e0c81 2023-03-30 03:59:01,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 03:59:01,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/210f1488269e4ef496060445fb761eda 2023-03-30 03:59:01,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/df110fe3b06d4e36aae8c7586b49e9a9 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/df110fe3b06d4e36aae8c7586b49e9a9 2023-03-30 03:59:01,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/df110fe3b06d4e36aae8c7586b49e9a9, entries=150, sequenceid=395, filesize=25.4 K 2023-03-30 03:59:01,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/c0163465c0384d99b3c91fe8ec4e0c81 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/c0163465c0384d99b3c91fe8ec4e0c81 2023-03-30 03:59:01,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/c0163465c0384d99b3c91fe8ec4e0c81, entries=150, sequenceid=395, filesize=11.9 K 2023-03-30 03:59:01,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/210f1488269e4ef496060445fb761eda as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/210f1488269e4ef496060445fb761eda 2023-03-30 03:59:01,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/210f1488269e4ef496060445fb761eda, entries=150, sequenceid=395, filesize=11.9 K 2023-03-30 03:59:01,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for 8ebabb5078c369476030f11b847f5c8f in 1685ms, sequenceid=395, compaction requested=true 2023-03-30 03:59:01,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:59:01,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:59:01,451 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. because compaction request was cancelled 2023-03-30 03:59:01,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:59:01,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 03:59:01,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. after waiting 0 ms 2023-03-30 03:59:01,451 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. because compaction request was cancelled 2023-03-30 03:59:01,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:59:01,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 03:59:01,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:59:01,452 DEBUG [RS:0;jenkins-hbase20:41157-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. because compaction request was cancelled 2023-03-30 03:59:01,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 8ebabb5078c369476030f11b847f5c8f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-30 03:59:01,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=A 2023-03-30 03:59:01,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:59:01,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=B 2023-03-30 03:59:01,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:59:01,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8ebabb5078c369476030f11b847f5c8f, store=C 2023-03-30 03:59:01,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 03:59:01,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303302aea60ccefb646f3ad087fd1b14b3f82 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302aea60ccefb646f3ad087fd1b14b3f82 2023-03-30 03:59:02,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 03:59:02,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=402, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/b1c5ff83f9ad4b7a96aa92a7dfa03c9a 2023-03-30 03:59:02,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/dc1f871b3dae4e7c83b7f2f2d013a210 2023-03-30 03:59:02,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/915a67e77a7c485bb3603002a155a74f 2023-03-30 03:59:02,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/A/b1c5ff83f9ad4b7a96aa92a7dfa03c9a as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/b1c5ff83f9ad4b7a96aa92a7dfa03c9a 2023-03-30 03:59:02,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/b1c5ff83f9ad4b7a96aa92a7dfa03c9a, entries=100, sequenceid=402, filesize=18.6 K 2023-03-30 03:59:02,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/B/dc1f871b3dae4e7c83b7f2f2d013a210 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/dc1f871b3dae4e7c83b7f2f2d013a210 2023-03-30 03:59:02,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/dc1f871b3dae4e7c83b7f2f2d013a210, entries=100, sequenceid=402, filesize=9.5 K 2023-03-30 03:59:02,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/.tmp/C/915a67e77a7c485bb3603002a155a74f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/915a67e77a7c485bb3603002a155a74f 2023-03-30 03:59:02,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/915a67e77a7c485bb3603002a155a74f, entries=100, sequenceid=402, filesize=9.5 K 2023-03-30 03:59:02,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 8ebabb5078c369476030f11b847f5c8f in 1345ms, sequenceid=402, compaction requested=true 2023-03-30 03:59:02,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/8831dfcb3e634cc4888cf0e470d25084, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/28adc2de37df4a4e90b264d8ff1b4a52, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/a789c3c22cab4682854c4520d674db91] to archive 2023-03-30 03:59:02,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:59:02,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/8831dfcb3e634cc4888cf0e470d25084 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/8831dfcb3e634cc4888cf0e470d25084 2023-03-30 03:59:02,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/28adc2de37df4a4e90b264d8ff1b4a52 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/28adc2de37df4a4e90b264d8ff1b4a52 2023-03-30 03:59:02,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/a789c3c22cab4682854c4520d674db91 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/a789c3c22cab4682854c4520d674db91 2023-03-30 03:59:02,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9392ca1a2fd7420eafcef68e4a62900d, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/692cc93ea70141919b217a7bda062c79, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/7d1de1a913e64233a154ae78d3f9ad92] to archive 2023-03-30 03:59:02,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:59:02,824 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9392ca1a2fd7420eafcef68e4a62900d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9392ca1a2fd7420eafcef68e4a62900d 2023-03-30 03:59:02,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/692cc93ea70141919b217a7bda062c79 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/692cc93ea70141919b217a7bda062c79 2023-03-30 03:59:02,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/7d1de1a913e64233a154ae78d3f9ad92 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/7d1de1a913e64233a154ae78d3f9ad92 2023-03-30 03:59:02,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/92d3d7532174495f975c339f54836e26, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/692b9450864040748d56a28589c56584, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/60d018c81d2c4b0687858c735cb7fb9d] to archive 2023-03-30 03:59:02,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 03:59:02,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/92d3d7532174495f975c339f54836e26 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/92d3d7532174495f975c339f54836e26 2023-03-30 03:59:02,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/692b9450864040748d56a28589c56584 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/692b9450864040748d56a28589c56584 2023-03-30 03:59:02,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/60d018c81d2c4b0687858c735cb7fb9d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/60d018c81d2c4b0687858c735cb7fb9d 2023-03-30 03:59:02,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/recovered.edits/405.seqid, newMaxSeqId=405, maxSeqId=4 2023-03-30 03:59:02,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f. 2023-03-30 03:59:02,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 8ebabb5078c369476030f11b847f5c8f: 2023-03-30 03:59:02,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:59:02,837 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=8ebabb5078c369476030f11b847f5c8f, regionState=CLOSED 2023-03-30 03:59:02,837 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680148742837"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680148742837"}]},"ts":"1680148742837"} 2023-03-30 03:59:02,840 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-30 03:59:02,840 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 8ebabb5078c369476030f11b847f5c8f, server=jenkins-hbase20.apache.org,41157,1680148568851 in 2.7030 sec 2023-03-30 03:59:02,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-30 03:59:02,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8ebabb5078c369476030f11b847f5c8f, UNASSIGN in 2.7070 sec 2023-03-30 03:59:02,842 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680148742842"}]},"ts":"1680148742842"} 2023-03-30 03:59:02,843 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 03:59:02,844 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 03:59:02,846 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.7190 sec 2023-03-30 03:59:04,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 03:59:04,241 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-30 03:59:04,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-30 03:59:04,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:59:04,244 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:59:04,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-30 03:59:04,245 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:59:04,249 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:59:04,251 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C, FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/recovered.edits] 2023-03-30 03:59:04,253 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/0fa743e083804dbc8b3ef0a4d4855739 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/0fa743e083804dbc8b3ef0a4d4855739 2023-03-30 03:59:04,254 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/11ec28b52ac9414bb6beef2b8edf20a6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/11ec28b52ac9414bb6beef2b8edf20a6 2023-03-30 03:59:04,255 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/2c08692d65804403b02cb1eecc7196c6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/2c08692d65804403b02cb1eecc7196c6 2023-03-30 03:59:04,256 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/2fe927f1a7024a5aae8ed781247d4ced to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/2fe927f1a7024a5aae8ed781247d4ced 2023-03-30 03:59:04,257 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/3ec193a48fc14688b50fcc4ea8ab2e33 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/3ec193a48fc14688b50fcc4ea8ab2e33 2023-03-30 03:59:04,258 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/699ec9a7438b4f17919cc5a61cc776da to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/699ec9a7438b4f17919cc5a61cc776da 2023-03-30 03:59:04,259 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/6e54e63fe0424375a7eee2cf0cde1f7d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/6e54e63fe0424375a7eee2cf0cde1f7d 2023-03-30 03:59:04,260 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/99b4030b8cad4fc0bf9e4b85509146c6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/99b4030b8cad4fc0bf9e4b85509146c6 2023-03-30 03:59:04,261 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/b1c5ff83f9ad4b7a96aa92a7dfa03c9a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/b1c5ff83f9ad4b7a96aa92a7dfa03c9a 2023-03-30 03:59:04,262 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/c980ab7546fc4299b112815d2c63ac49 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/c980ab7546fc4299b112815d2c63ac49 2023-03-30 03:59:04,263 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/c9f4df2c90c047fb83663838d3ab8229 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/c9f4df2c90c047fb83663838d3ab8229 2023-03-30 03:59:04,264 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/cac7cb410480471e851a410c3bd8c382 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/cac7cb410480471e851a410c3bd8c382 2023-03-30 03:59:04,265 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/d25001fcb77c4da197fe647a9363c81c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/d25001fcb77c4da197fe647a9363c81c 2023-03-30 03:59:04,266 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/df110fe3b06d4e36aae8c7586b49e9a9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/df110fe3b06d4e36aae8c7586b49e9a9 2023-03-30 03:59:04,267 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/ee30d47787a04d639c20f58fc97cd306 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/ee30d47787a04d639c20f58fc97cd306 2023-03-30 03:59:04,268 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/ff7545dea33042e5acf347286c376fb5 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/A/ff7545dea33042e5acf347286c376fb5 2023-03-30 03:59:04,270 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/056740f697044d8bbcceea0cd32b42c9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/056740f697044d8bbcceea0cd32b42c9 2023-03-30 03:59:04,271 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/3d059ea5612a4eb2a467fb0ab0546801 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/3d059ea5612a4eb2a467fb0ab0546801 2023-03-30 03:59:04,273 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/404ad2404f65429fb8946ade1b88d8de to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/404ad2404f65429fb8946ade1b88d8de 2023-03-30 03:59:04,274 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/41acef4f506442cdaff056ce6c74935c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/41acef4f506442cdaff056ce6c74935c 2023-03-30 03:59:04,275 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/53fe4a1e256b4650be9c405397156a83 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/53fe4a1e256b4650be9c405397156a83 2023-03-30 03:59:04,276 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/5c668ec77e074dd9a78ad1e12b53e9eb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/5c668ec77e074dd9a78ad1e12b53e9eb 2023-03-30 03:59:04,277 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/5f93034e5d5746d2aaad3f7358c0a2c8 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/5f93034e5d5746d2aaad3f7358c0a2c8 2023-03-30 03:59:04,278 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/67b9be0d3ca24c0e863f0233c5658141 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/67b9be0d3ca24c0e863f0233c5658141 2023-03-30 03:59:04,279 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/6b6016bdb888430f990a0e7c52192d87 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/6b6016bdb888430f990a0e7c52192d87 2023-03-30 03:59:04,279 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9851d16222944c3eafeb7851bd36a204 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/9851d16222944c3eafeb7851bd36a204 2023-03-30 03:59:04,280 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/98e7e88713a645568b029378b88de9ec to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/98e7e88713a645568b029378b88de9ec 2023-03-30 03:59:04,281 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/a4c19ba128774157af80739c8f0b9fc7 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/a4c19ba128774157af80739c8f0b9fc7 2023-03-30 03:59:04,281 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/b6d5c7734f494cedb4b9efd6c39c5371 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/b6d5c7734f494cedb4b9efd6c39c5371 2023-03-30 03:59:04,282 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/c0163465c0384d99b3c91fe8ec4e0c81 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/c0163465c0384d99b3c91fe8ec4e0c81 2023-03-30 03:59:04,283 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/c0f510b4a1bf4163b371a09e82491a91 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/c0f510b4a1bf4163b371a09e82491a91 2023-03-30 03:59:04,283 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/dc1f871b3dae4e7c83b7f2f2d013a210 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/B/dc1f871b3dae4e7c83b7f2f2d013a210 2023-03-30 03:59:04,285 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/0d398f1e53794a2781fa52cf81a4549b to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/0d398f1e53794a2781fa52cf81a4549b 2023-03-30 03:59:04,285 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/1f987d992b044e2697c427f4373c23bb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/1f987d992b044e2697c427f4373c23bb 2023-03-30 03:59:04,286 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/210f1488269e4ef496060445fb761eda to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/210f1488269e4ef496060445fb761eda 2023-03-30 03:59:04,287 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/382da7500dcd406d9e9b83bd63dc5b61 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/382da7500dcd406d9e9b83bd63dc5b61 2023-03-30 03:59:04,287 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/3a5ba214014741e1bfcd60d17330c515 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/3a5ba214014741e1bfcd60d17330c515 2023-03-30 03:59:04,288 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/418da646af10401586b93dc37b8ff7c6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/418da646af10401586b93dc37b8ff7c6 2023-03-30 03:59:04,289 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/521548cb9ac54be6830065efc7b0cec3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/521548cb9ac54be6830065efc7b0cec3 2023-03-30 03:59:04,289 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/5369c341305c41cfb23d79d930aa504d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/5369c341305c41cfb23d79d930aa504d 2023-03-30 03:59:04,290 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/743cb74560034b619da34af69c201b19 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/743cb74560034b619da34af69c201b19 2023-03-30 03:59:04,291 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/7bc87927311045bd90c9abf78b99184a to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/7bc87927311045bd90c9abf78b99184a 2023-03-30 03:59:04,292 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/89fe83e00f854e34bd816c0ea5795564 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/89fe83e00f854e34bd816c0ea5795564 2023-03-30 03:59:04,292 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/8d27ce006eb14c1cb8f3d0c5b859434d to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/8d27ce006eb14c1cb8f3d0c5b859434d 2023-03-30 03:59:04,293 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/8edfd59c927a4ff898fa7d2f4210cd99 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/8edfd59c927a4ff898fa7d2f4210cd99 2023-03-30 03:59:04,294 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/915a67e77a7c485bb3603002a155a74f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/915a67e77a7c485bb3603002a155a74f 2023-03-30 03:59:04,294 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/91a3cbd7e03d444db567319685eee7fb to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/91a3cbd7e03d444db567319685eee7fb 2023-03-30 03:59:04,295 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/bac69369db3847168e8530ced9a5d3ed to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/C/bac69369db3847168e8530ced9a5d3ed 2023-03-30 03:59:04,296 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/recovered.edits/405.seqid to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f/recovered.edits/405.seqid 2023-03-30 03:59:04,297 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/.tmp/data/default/TestAcidGuarantees/8ebabb5078c369476030f11b847f5c8f 2023-03-30 03:59:04,297 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 03:59:04,297 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 03:59:04,297 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-30 03:59:04,299 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301de8f94216a94f8489fce444e305b85c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301de8f94216a94f8489fce444e305b85c 2023-03-30 03:59:04,300 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302a0e962f199347738aaddfc5ee6b29b3 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302a0e962f199347738aaddfc5ee6b29b3 2023-03-30 03:59:04,300 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302aea60ccefb646f3ad087fd1b14b3f82 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302aea60ccefb646f3ad087fd1b14b3f82 2023-03-30 03:59:04,301 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303ab7682ea4684d0c8537f9b8797993b9 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303ab7682ea4684d0c8537f9b8797993b9 2023-03-30 03:59:04,302 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033051173181a905435584af096883297bc2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033051173181a905435584af096883297bc2 2023-03-30 03:59:04,303 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033057d2c40410fa47fe9fa3e82ef018465f to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033057d2c40410fa47fe9fa3e82ef018465f 2023-03-30 03:59:04,304 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306188898d37624e219035ace8543f7664 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306188898d37624e219035ace8543f7664 2023-03-30 03:59:04,304 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330836921daabdf4415bee7b19464073e06 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330836921daabdf4415bee7b19464073e06 2023-03-30 03:59:04,305 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308563ee47e7474122b540a34d3acfaa37 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308563ee47e7474122b540a34d3acfaa37 2023-03-30 03:59:04,306 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308f9d381296684fc3a22d02b162aa1729 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308f9d381296684fc3a22d02b162aa1729 2023-03-30 03:59:04,307 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033093b1fc6658294b27be3a229cff239df2 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033093b1fc6658294b27be3a229cff239df2 2023-03-30 03:59:04,308 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309530c105b6c84b7b805f136798ffc93c to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309530c105b6c84b7b805f136798ffc93c 2023-03-30 03:59:04,309 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309e76eac841304ec890b2e71e4b9d6c46 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309e76eac841304ec890b2e71e4b9d6c46 2023-03-30 03:59:04,310 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309fd922802b98472d8319f7786dc1fcc7 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309fd922802b98472d8319f7786dc1fcc7 2023-03-30 03:59:04,311 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330ad1f0da5f57549018724aa546bc0f7c6 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330ad1f0da5f57549018724aa546bc0f7c6 2023-03-30 03:59:04,312 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b6ee51cf310e4f9aa7f22007f3026311 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b6ee51cf310e4f9aa7f22007f3026311 2023-03-30 03:59:04,313 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d314415daece41af98d0ebd8252ced18 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d314415daece41af98d0ebd8252ced18 2023-03-30 03:59:04,314 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f536cd3588584f798fc6104b2fdba3e4 to hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f536cd3588584f798fc6104b2fdba3e4 2023-03-30 03:59:04,314 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 03:59:04,316 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:59:04,318 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 03:59:04,319 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 03:59:04,320 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:59:04,320 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 03:59:04,320 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680148744320"}]},"ts":"9223372036854775807"} 2023-03-30 03:59:04,322 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 03:59:04,322 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8ebabb5078c369476030f11b847f5c8f, NAME => 'TestAcidGuarantees,,1680148716064.8ebabb5078c369476030f11b847f5c8f.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 03:59:04,322 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 03:59:04,323 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680148744322"}]},"ts":"9223372036854775807"} 2023-03-30 03:59:04,324 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 03:59:04,326 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 03:59:04,327 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 84 msec 2023-03-30 03:59:04,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37051] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-30 03:59:04,346 INFO [Listener at localhost.localdomain/37215] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-30 03:59:04,363 INFO [Listener at localhost.localdomain/37215] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=516 (was 524), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=381 (was 458), ProcessCount=166 (was 166), AvailableMemoryMB=11100 (was 11071) - AvailableMemoryMB LEAK? - 2023-03-30 03:59:04,364 INFO [Listener at localhost.localdomain/37215] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-30 03:59:04,364 INFO [Listener at localhost.localdomain/37215] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 03:59:04,364 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dc6e9d6 to 127.0.0.1:58033 2023-03-30 03:59:04,364 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:59:04,364 DEBUG [Listener at localhost.localdomain/37215] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-30 03:59:04,365 DEBUG [Listener at localhost.localdomain/37215] util.JVMClusterUtil(257): Found active master hash=1506056166, stopped=false 2023-03-30 03:59:04,365 INFO [Listener at localhost.localdomain/37215] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:59:04,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-30 03:59:04,365 INFO [Listener at localhost.localdomain/37215] procedure2.ProcedureExecutor(629): Stopping 2023-03-30 03:59:04,365 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-30 03:59:04,366 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:59:04,366 DEBUG [Listener at localhost.localdomain/37215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5adf05a0 to 127.0.0.1:58033 2023-03-30 03:59:04,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-30 03:59:04,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-30 03:59:04,366 DEBUG [Listener at localhost.localdomain/37215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:59:04,366 INFO [Listener at localhost.localdomain/37215] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,41157,1680148568851' ***** 2023-03-30 03:59:04,366 INFO [Listener at localhost.localdomain/37215] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-30 03:59:04,367 INFO [RS:0;jenkins-hbase20:41157] regionserver.HeapMemoryManager(220): Stopping 2023-03-30 03:59:04,367 INFO [RS:0;jenkins-hbase20:41157] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-30 03:59:04,367 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-30 03:59:04,367 INFO [RS:0;jenkins-hbase20:41157] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-30 03:59:04,367 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(3304): Received CLOSE for 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:59:04,368 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:59:04,368 DEBUG [RS:0;jenkins-hbase20:41157] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35a717e9 to 127.0.0.1:58033 2023-03-30 03:59:04,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 4f7ceee8cd66bf2f0907e201f9289fa0, disabling compactions & flushes 2023-03-30 03:59:04,368 DEBUG [RS:0;jenkins-hbase20:41157] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:59:04,368 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:59:04,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:59:04,368 INFO [RS:0;jenkins-hbase20:41157] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-30 03:59:04,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. after waiting 0 ms 2023-03-30 03:59:04,369 INFO [RS:0;jenkins-hbase20:41157] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-30 03:59:04,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:59:04,369 INFO [RS:0;jenkins-hbase20:41157] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-30 03:59:04,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 4f7ceee8cd66bf2f0907e201f9289fa0 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-30 03:59:04,369 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-30 03:59:04,369 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-30 03:59:04,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-30 03:59:04,370 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-30 03:59:04,370 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 4f7ceee8cd66bf2f0907e201f9289fa0=hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0.} 2023-03-30 03:59:04,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-30 03:59:04,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-30 03:59:04,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-30 03:59:04,370 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-03-30 03:59:04,371 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1505): Waiting on 1588230740, 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:59:04,424 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-30 03:59:04,487 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-30 03:59:04,487 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-30 03:59:04,572 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1505): Waiting on 1588230740, 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:59:04,772 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1505): Waiting on 1588230740, 4f7ceee8cd66bf2f0907e201f9289fa0 2023-03-30 03:59:04,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0/.tmp/info/3e088edeaa5244ee9e0b7f370cb2d2e8 2023-03-30 03:59:04,783 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/.tmp/info/c74e3801b695437b912e0402cc73e04f 2023-03-30 03:59:04,787 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c74e3801b695437b912e0402cc73e04f 2023-03-30 03:59:04,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0/.tmp/info/3e088edeaa5244ee9e0b7f370cb2d2e8 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0/info/3e088edeaa5244ee9e0b7f370cb2d2e8 2023-03-30 03:59:04,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0/info/3e088edeaa5244ee9e0b7f370cb2d2e8, entries=2, sequenceid=6, filesize=4.8 K 2023-03-30 03:59:04,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 4f7ceee8cd66bf2f0907e201f9289fa0 in 423ms, sequenceid=6, compaction requested=false 2023-03-30 03:59:04,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/namespace/4f7ceee8cd66bf2f0907e201f9289fa0/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-30 03:59:04,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:59:04,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 4f7ceee8cd66bf2f0907e201f9289fa0: 2023-03-30 03:59:04,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680148571540.4f7ceee8cd66bf2f0907e201f9289fa0. 2023-03-30 03:59:04,973 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-30 03:59:05,173 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-30 03:59:05,194 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/.tmp/rep_barrier/e4af9d93a54d401b8e8ee693838009cd 2023-03-30 03:59:05,204 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e4af9d93a54d401b8e8ee693838009cd 2023-03-30 03:59:05,373 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-30 03:59:05,374 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-03-30 03:59:05,374 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-30 03:59:05,574 DEBUG [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-30 03:59:05,613 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/.tmp/table/0269c3f2835a4e99924e5f17c092a430 2023-03-30 03:59:05,616 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0269c3f2835a4e99924e5f17c092a430 2023-03-30 03:59:05,617 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/.tmp/info/c74e3801b695437b912e0402cc73e04f as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/info/c74e3801b695437b912e0402cc73e04f 2023-03-30 03:59:05,620 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c74e3801b695437b912e0402cc73e04f 2023-03-30 03:59:05,620 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/info/c74e3801b695437b912e0402cc73e04f, entries=22, sequenceid=93, filesize=7.1 K 2023-03-30 03:59:05,621 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/.tmp/rep_barrier/e4af9d93a54d401b8e8ee693838009cd as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/rep_barrier/e4af9d93a54d401b8e8ee693838009cd 2023-03-30 03:59:05,624 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e4af9d93a54d401b8e8ee693838009cd 2023-03-30 03:59:05,624 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/rep_barrier/e4af9d93a54d401b8e8ee693838009cd, entries=6, sequenceid=93, filesize=5.5 K 2023-03-30 03:59:05,624 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/.tmp/table/0269c3f2835a4e99924e5f17c092a430 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/table/0269c3f2835a4e99924e5f17c092a430 2023-03-30 03:59:05,627 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0269c3f2835a4e99924e5f17c092a430 2023-03-30 03:59:05,627 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/table/0269c3f2835a4e99924e5f17c092a430, entries=9, sequenceid=93, filesize=5.5 K 2023-03-30 03:59:05,628 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 1258ms, sequenceid=93, compaction requested=false 2023-03-30 03:59:05,636 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-30 03:59:05,637 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-30 03:59:05,638 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-30 03:59:05,638 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-30 03:59:05,638 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-30 03:59:05,774 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,41157,1680148568851; all regions closed. 2023-03-30 03:59:05,789 DEBUG [RS:0;jenkins-hbase20:41157] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/oldWALs 2023-03-30 03:59:05,789 INFO [RS:0;jenkins-hbase20:41157] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C41157%2C1680148568851.meta:.meta(num 1680148571255) 2023-03-30 03:59:05,795 DEBUG [RS:0;jenkins-hbase20:41157] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/oldWALs 2023-03-30 03:59:05,795 INFO [RS:0;jenkins-hbase20:41157] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C41157%2C1680148568851:(num 1680148570653) 2023-03-30 03:59:05,795 DEBUG [RS:0;jenkins-hbase20:41157] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:59:05,795 INFO [RS:0;jenkins-hbase20:41157] regionserver.LeaseManager(133): Closed leases 2023-03-30 03:59:05,795 INFO [RS:0;jenkins-hbase20:41157] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-03-30 03:59:05,795 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-30 03:59:05,796 INFO [RS:0;jenkins-hbase20:41157] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:41157 2023-03-30 03:59:05,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,41157,1680148568851 2023-03-30 03:59:05,801 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-30 03:59:05,800 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-30 03:59:05,801 ERROR [Listener at localhost.localdomain/37215-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6417af3b rejected from java.util.concurrent.ThreadPoolExecutor@60282761[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2297] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 03:59:05,801 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,41157,1680148568851] 2023-03-30 03:59:05,802 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,41157,1680148568851; numProcessing=1 2023-03-30 03:59:05,803 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,41157,1680148568851 already deleted, retry=false 2023-03-30 03:59:05,803 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,41157,1680148568851 expired; onlineServers=0 2023-03-30 03:59:05,803 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,37051,1680148567795' ***** 2023-03-30 03:59:05,803 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-30 03:59:05,803 DEBUG [M:0;jenkins-hbase20:37051] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18f99ddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-30 03:59:05,803 INFO [M:0;jenkins-hbase20:37051] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,37051,1680148567795 2023-03-30 03:59:05,803 INFO [M:0;jenkins-hbase20:37051] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,37051,1680148567795; all regions closed. 2023-03-30 03:59:05,803 DEBUG [M:0;jenkins-hbase20:37051] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 03:59:05,803 DEBUG [M:0;jenkins-hbase20:37051] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-30 03:59:05,803 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-30 03:59:05,803 DEBUG [M:0;jenkins-hbase20:37051] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-30 03:59:05,803 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680148570312] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680148570312,5,FailOnTimeoutGroup] 2023-03-30 03:59:05,803 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680148570318] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680148570318,5,FailOnTimeoutGroup] 2023-03-30 03:59:05,803 INFO [M:0;jenkins-hbase20:37051] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-30 03:59:05,803 INFO [M:0;jenkins-hbase20:37051] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-30 03:59:05,803 INFO [M:0;jenkins-hbase20:37051] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-03-30 03:59:05,803 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-30 03:59:05,804 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 03:59:05,804 DEBUG [M:0;jenkins-hbase20:37051] master.HMaster(1512): Stopping service threads 2023-03-30 03:59:05,804 INFO [M:0;jenkins-hbase20:37051] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-30 03:59:05,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-30 03:59:05,804 ERROR [M:0;jenkins-hbase20:37051] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (207162374) connection to localhost.localdomain/127.0.0.1:36859 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-30 03:59:05,805 INFO [M:0;jenkins-hbase20:37051] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-30 03:59:05,805 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-30 03:59:05,805 DEBUG [M:0;jenkins-hbase20:37051] zookeeper.ZKUtil(398): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-30 03:59:05,806 WARN [M:0;jenkins-hbase20:37051] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-30 03:59:05,806 INFO [M:0;jenkins-hbase20:37051] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-30 03:59:05,806 INFO [M:0;jenkins-hbase20:37051] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-30 03:59:05,806 DEBUG [M:0;jenkins-hbase20:37051] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-30 03:59:05,807 INFO [M:0;jenkins-hbase20:37051] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 03:59:05,807 DEBUG [M:0;jenkins-hbase20:37051] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 03:59:05,807 DEBUG [M:0;jenkins-hbase20:37051] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-30 03:59:05,807 DEBUG [M:0;jenkins-hbase20:37051] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 03:59:05,807 INFO [M:0;jenkins-hbase20:37051] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.17 KB heapSize=393.88 KB 2023-03-30 03:59:05,902 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-30 03:59:05,902 INFO [RS:0;jenkins-hbase20:41157] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,41157,1680148568851; zookeeper connection closed. 2023-03-30 03:59:05,902 ERROR [Listener at localhost.localdomain/37215-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4df2d570 rejected from java.util.concurrent.ThreadPoolExecutor@60282761[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2297] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 03:59:05,903 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@10f483c8] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@10f483c8 2023-03-30 03:59:05,903 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): regionserver:41157-0x100605eabcf0001, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-30 03:59:05,903 ERROR [Listener at localhost.localdomain/37215-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3577eb44 rejected from java.util.concurrent.ThreadPoolExecutor@60282761[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2297] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 03:59:05,904 INFO [Listener at localhost.localdomain/37215] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-30 03:59:06,227 INFO [M:0;jenkins-hbase20:37051] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.17 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e216ba77c9b0454f8ddc377eea038cc0 2023-03-30 03:59:06,238 INFO [M:0;jenkins-hbase20:37051] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e216ba77c9b0454f8ddc377eea038cc0 2023-03-30 03:59:06,239 DEBUG [M:0;jenkins-hbase20:37051] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e216ba77c9b0454f8ddc377eea038cc0 as hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e216ba77c9b0454f8ddc377eea038cc0 2023-03-30 03:59:06,245 INFO [M:0;jenkins-hbase20:37051] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e216ba77c9b0454f8ddc377eea038cc0 2023-03-30 03:59:06,246 INFO [M:0;jenkins-hbase20:37051] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36859/user/jenkins/test-data/be746f29-3517-95cb-b522-2c3f55137f95/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e216ba77c9b0454f8ddc377eea038cc0, entries=65, sequenceid=578, filesize=24.7 K 2023-03-30 03:59:06,246 INFO [M:0;jenkins-hbase20:37051] regionserver.HRegion(2948): Finished flush of dataSize ~345.17 KB/353451, heapSize ~393.87 KB/403320, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 439ms, sequenceid=578, compaction requested=false 2023-03-30 03:59:06,248 INFO [M:0;jenkins-hbase20:37051] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 03:59:06,248 DEBUG [M:0;jenkins-hbase20:37051] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-30 03:59:06,252 INFO [M:0;jenkins-hbase20:37051] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-30 03:59:06,252 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-30 03:59:06,253 INFO [M:0;jenkins-hbase20:37051] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:37051 2023-03-30 03:59:06,254 DEBUG [M:0;jenkins-hbase20:37051] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,37051,1680148567795 already deleted, retry=false 2023-03-30 03:59:06,356 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-30 03:59:06,356 INFO [M:0;jenkins-hbase20:37051] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,37051,1680148567795; zookeeper connection closed. 2023-03-30 03:59:06,356 ERROR [Listener at localhost.localdomain/37215-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@d4fefbd rejected from java.util.concurrent.ThreadPoolExecutor@2c77132c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3842] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 03:59:06,358 DEBUG [Listener at localhost.localdomain/37215-EventThread] zookeeper.ZKWatcher(600): master:37051-0x100605eabcf0000, quorum=127.0.0.1:58033, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-30 03:59:06,358 ERROR [Listener at localhost.localdomain/37215-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@51c94c86 rejected from java.util.concurrent.ThreadPoolExecutor@2c77132c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3842] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 03:59:06,361 WARN [Listener at localhost.localdomain/37215] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-30 03:59:06,366 INFO [Listener at localhost.localdomain/37215] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-30 03:59:06,481 WARN [BP-172349167-148.251.75.209-1680148564762 heartbeating to localhost.localdomain/127.0.0.1:36859] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-30 03:59:06,481 WARN [BP-172349167-148.251.75.209-1680148564762 heartbeating to localhost.localdomain/127.0.0.1:36859] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-172349167-148.251.75.209-1680148564762 (Datanode Uuid 00825c34-6b27-4d8f-a0b1-9ef71ed73366) service to localhost.localdomain/127.0.0.1:36859 2023-03-30 03:59:06,484 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/cluster_93f5cab8-c930-d024-2758-c8a21862fe6f/dfs/data/data1/current/BP-172349167-148.251.75.209-1680148564762] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-30 03:59:06,485 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894ebd2c-e32c-4b51-7892-7ed6a2ee0c4e/cluster_93f5cab8-c930-d024-2758-c8a21862fe6f/dfs/data/data2/current/BP-172349167-148.251.75.209-1680148564762] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-30 03:59:06,521 INFO [Listener at localhost.localdomain/37215] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-03-30 03:59:06,677 INFO [Listener at localhost.localdomain/37215] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-30 03:59:06,830 INFO [Listener at localhost.localdomain/37215] hbase.HBaseTestingUtility(1293): Minicluster is down